Refresh native configuration, CLI, and LLM internal guides
This commit is contained in:
@@ -1,161 +1,148 @@
|
||||
# Configuration Internals
|
||||
|
||||
This document describes the maintainer-facing configuration boundary in
|
||||
**internal/core/config**. The [Configuration](../config.md) reference owns the
|
||||
file format, fields, defaults, precedence contract, and selectable keys. The
|
||||
[CLI reference](../cli.md) owns command syntax; this document does not redefine
|
||||
either interface.
|
||||
This guide explains the maintained configuration boundary and its composition-root
|
||||
handoff. [Configuration](../config.md) owns source selection, syntax, fields,
|
||||
defaults, and selectable modules; [CLI Reference](../cli.md) owns invocation.
|
||||
[Architecture](../policy/architecture.md#configuration-and-provenance) defines
|
||||
ownership. Do not infer operator contracts from private helper choices here.
|
||||
|
||||
## Boundary
|
||||
## Owners And Handoffs
|
||||
|
||||
The configuration package turns a selected ConfigKit-composed YAML document
|
||||
into a validated, independently owned configuration. It then resolves a
|
||||
requested pipeline against a module catalog before the framework prepares or
|
||||
runs anything.
|
||||
| Owner | Responsibility |
|
||||
| --- | --- |
|
||||
| [core/config definition](../../internal/core/config/load.go) | Typed application document, native PromptKit configuration rules, application resolution/validation. |
|
||||
| [CLI loaders](../../internal/cli/configkit_load.go) | Main-source selection, native underlays, invocation-local result/engine reuse. |
|
||||
| [LoadedConfig](../../internal/core/config/loaded_config.go) | Retain the original result and derive independently owned runtime pipeline configuration. |
|
||||
| [Config.Resolve](../../internal/core/config/effective_config.go) | Resolve a selected pipeline against the supplied catalog. |
|
||||
| [preflight](../../internal/preflight/preflight.go) | Check known native resources from resolved composition and supplied engine. |
|
||||
| Pipeline preparation | Construct typed execution collaborators from resolved, materialized inputs; see [Pipeline Internals](pipeline.md). |
|
||||
|
||||
| Boundary | Inputs | Outputs | Does not own |
|
||||
| --- | --- | --- | --- |
|
||||
| Loading | Selected ConfigKit source, overlay, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the source or reporting a command result. |
|
||||
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
|
||||
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
|
||||
| Inspection | **LoadedConfig** and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. |
|
||||
The production flow keeps one application-owned
|
||||
`*configkit.Result[config.Document]`. That result feeds runtime conversion,
|
||||
whole-app/focused inspection, and `promptkit.NewFromConfig`. Rendered YAML is
|
||||
an operator view, not an interchange format for constructing the engine.
|
||||
|
||||
The CLI describes the selected source, invokes this package, and supplies the
|
||||
result to the framework. Configuration never reads an input file, constructs a
|
||||
module, or creates output, cache, or debug paths. Those responsibilities remain
|
||||
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
|
||||
[run-state](state.md) boundaries.
|
||||
## Definition And Application Meaning
|
||||
|
||||
## Loading And Validation
|
||||
`Document` contains native `promptkit.Settings` at its PromptKit field.
|
||||
`Definition` composes application defaults, version policy, sensitivity,
|
||||
identified step lists, resolution, and validation with `promptkit.Configure`.
|
||||
Native resource rules therefore participate in the same load rather than being
|
||||
validated by a separate application parser.
|
||||
|
||||
`Load` selects an exclusive file; `LoadSource` accepts the CLI's source and
|
||||
enables imports through the same definition and callback path. ConfigKit composes
|
||||
the main input, imports, and optional overlay, then expands native inheritance
|
||||
before typed decoding. Explicit CLI assignments follow decoding and precede
|
||||
application resolution. The document definition supplies typed
|
||||
defaults and structural validation; Notarius's recorded resolution changes
|
||||
normalize configured paths after decode. Reference paths resolve before key
|
||||
normalization moves their source coordinates. Ordinary list leaf edits are
|
||||
recorded at the leaf; a changed step identity requires a covering list record.
|
||||
Actual map-key moves retain structural records. Path normalization follows the source
|
||||
of the surviving value, including a value copied through inheritance, rather
|
||||
than the inheritance directive's location. Inherited input remains explicit
|
||||
for presence-sensitive behavior, including zero, false, and empty values;
|
||||
existing field validation still governs whether the resulting values are valid.
|
||||
ConfigKit's inspection history retains inheritance transfers alongside later
|
||||
normalization, and effective configuration renders the normalized values used
|
||||
by runtime conversion. The private typed document consumes the directive, so
|
||||
no upstream inheritance type or unresolved directive reaches runtime pipeline
|
||||
or module contracts. See [Configuration](../config.md#sibling-inheritance)
|
||||
for the operator syntax and field rules. Only
|
||||
`NOTARIUS_CONFIG` participates in discovery; retired field-level environment
|
||||
variables do not participate in loading. Version 5 decoding rejects scalar
|
||||
module bindings and scalar external references. ConfigKit retains redacted,
|
||||
historical inspection metadata while **LoadedConfig** keeps that implementation
|
||||
detail private behind effective and explanation writers. `MainSourcePath` reads
|
||||
the non-imported main file from retained sources, and runtime conversion stays detached.
|
||||
Application callbacks normalize pipeline identifiers and operational paths while
|
||||
retaining presence/source history. The per-load state tracks original coordinates
|
||||
as map keys move. Paths are resolved before normalization loses their source
|
||||
coordinates. Inherited values retain their contributing origins; CLI path
|
||||
assignments use invocation origin. See
|
||||
[document paths](../../internal/core/config/document_paths.go) and
|
||||
[pipeline document normalization](../../internal/core/config/pipeline_document.go).
|
||||
|
||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||
concurrency settings, malformed bindings and references, invalid retries, and
|
||||
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||
information, query, or fragment, and rejects a negative local concurrency
|
||||
limit. It also rejects a negative `promptkit.transient_retries` value; decoded
|
||||
omission resolves to two, while explicit zero remains a distinct opt-out. Its
|
||||
errors retain the closest known pipeline, lane, and binding context.
|
||||
It deliberately does not require modules to be registered: that requires a
|
||||
catalog and belongs to resolution.
|
||||
ConfigKit owns parsing, imports, overlays, inheritance, typed composition, and
|
||||
provenance. The application supplies meaning through supported rules. It does
|
||||
not implement another inheritance resolver or translate retired inputs.
|
||||
[The public reference](../config.md#imports-and-sibling-inheritance) owns exact
|
||||
presence/null/list rules and processing order.
|
||||
|
||||
The exact user-selectable values and validation rules are defined in
|
||||
[Configuration](../config.md). Keep document schema, source-aware normalization,
|
||||
runtime conversion, and that reference aligned in the same change.
|
||||
`Config.Validate` checks application-only state and pipeline structure without
|
||||
requiring registered modules. Catalog-dependent compatibility belongs to
|
||||
resolution. Configuration does not parse a source transcript, materialize
|
||||
references, construct modules, or allocate run storage.
|
||||
|
||||
## Effective Resolution
|
||||
## Production Source Assembly
|
||||
|
||||
**Config.Resolve** derives an absent extraction-worker value in an owned copy
|
||||
and validates the configuration. A programmatically supplied worker value is
|
||||
always explicit. It normalizes the requested pipeline ID, copies the selected
|
||||
profile, and passes the non-empty command-level LLM profile override, requested
|
||||
lane selection, and reference changes to the framework resolver.
|
||||
`nativeLoaders.application` selects the main source and uses
|
||||
`configkit.LoadLayered` with the application definition, native underlays,
|
||||
selected overlay, and explicit overrides. Underlays come from bundled provider
|
||||
resources, ordinary embedded application content, and main-relative operator
|
||||
companions. Native source/filesystem contexts survive the load.
|
||||
|
||||
After module and validator selection, the resolver applies the effective
|
||||
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||
profile-free, and no second inheritance decision occurs during execution. The
|
||||
public field definitions and precedence are owned by
|
||||
[Configuration](../config.md#pipelines).
|
||||
The embedded assets package supplies content, not registration behavior.
|
||||
The operator companion uses a native main-relative context; it follows the
|
||||
selected main rather than rediscovering the working directory. Strict peers,
|
||||
optional absence, imports/destinations, and whole-record resource replacement
|
||||
remain native composition. Source grammar and precedence belong in
|
||||
[Configuration](../config.md#configuration-discovery-and-precedence).
|
||||
|
||||
The resolver retains configured `validation_policy` overrides and derives one
|
||||
detached concrete terminal policy for the chunk producer and every lane's
|
||||
extract, merge, and normalize producers. That field-by-field inheritance is
|
||||
complete before preparation, and the effective values contribute to pipeline
|
||||
and checkpoint identity; execution does not interpret configuration defaults.
|
||||
Core `Load`/`LoadSource` entry points support selected sources and the same
|
||||
definition/conversion for direct consumers and tests. They do not replace the
|
||||
production composition root's underlay assembly. Do not accidentally introduce
|
||||
a second production load by combining those paths.
|
||||
|
||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||
chains, checks registered module and artifact compatibility, validates module
|
||||
options, and returns the fixed ordered pipeline shape. Positive validator retry
|
||||
budgets require an LLM-backed selected validator; deterministic validators are
|
||||
rejected during resolution. Eligible LLM-backed producer specifications also
|
||||
contribute their declared correction protocol to the resolved metadata. The
|
||||
resulting **EffectiveConfig** retains the selected ID, requested selection and reference
|
||||
changes, a clone of the input configuration, and the resolved pipeline.
|
||||
Callers may therefore retain or modify their input slices and maps without
|
||||
changing the resolved result, and later consumers cannot mutate the original
|
||||
configuration through the effective value. This ownership includes the nested
|
||||
PromptKit local-backend value.
|
||||
## Result Lifetime And Runtime Conversion
|
||||
|
||||
Resolution failures stop before module construction and source parsing. They
|
||||
include an error path for an unconfigured pipeline, missing module, missing
|
||||
capability, incompatible artifact variant, invalid option, invalid reference,
|
||||
or invalid lane selection. CLI code maps these valid-invocation failures to the
|
||||
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
|
||||
`LoadedConfig.Result()` exposes the original typed result for intentional native
|
||||
handoff. `FromResult` derives pipeline runtime settings without loading sources
|
||||
or serializing settings again. `RuntimeConfig` returns a detached copy;
|
||||
the native PromptKit configuration stays with its owning result/engine rather
|
||||
than being copied into a private runtime wrapper.
|
||||
|
||||
## Resolved Identity And Redaction
|
||||
`MainSourcePath` reads the retained non-imported main-file evidence. It does not
|
||||
repeat discovery or assume the first acquired file is the main.
|
||||
|
||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||
after defaults, lane selection, module bindings, reference bindings, validator
|
||||
chains, selected correction protocols, effective LLM profiles, and artifact
|
||||
schema identity have been resolved. The digest excludes
|
||||
its own stored value. It identifies resolved composition rather than raw YAML
|
||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||
provenance before execution; cache and checkpoint identity have additional
|
||||
owners in [Run State Internals](state.md).
|
||||
Within one invocation, `nativeLoaders` retains results by complete
|
||||
`ckconfig.Selection`, including overlay and override presence/values. Captured
|
||||
override containers are copied. A repeated identical selection returns the
|
||||
same result; distinct diff sides remain distinct selections. The engine is
|
||||
constructed lazily once for its retained result. This is invocation state,
|
||||
not a process-wide persistent loader cache.
|
||||
|
||||
Configuration publication uses ConfigKit's redacted effective YAML. Resolved
|
||||
pipeline publication uses **RedactedResolvedPipelinePayload**, which copies
|
||||
bindings and nested option containers, replaces credential-shaped values with
|
||||
**[REDACTED]**, and omits materialized reference content while retaining safe
|
||||
provenance. This redaction is deliberately narrow and does not authorize
|
||||
recording arbitrary environment values or provider requests.
|
||||
`NewFromConfig` consumes the completed result, preserving native contexts and
|
||||
inspection evidence without rerunning callbacks or adding underlays. Runtime-only
|
||||
HTTP/client settings and test provider options stay at engine construction.
|
||||
[PromptKit's application-owned-load contract][handoff] owns the library API.
|
||||
|
||||
## Invariants To Preserve
|
||||
## Effective Resolution And Preflight
|
||||
|
||||
- ConfigKit composition applies defaults, imports, an optional overlay, and
|
||||
explicit overrides in one direction with retained origin metadata.
|
||||
- A configuration is structurally valid before it is resolved, and a resolved
|
||||
pipeline is compatible with the supplied catalog before preparation begins.
|
||||
- Whitespace-normalized identifiers are unique wherever they identify a
|
||||
pipeline, step, lane, worker, or reference slot.
|
||||
- Runtime conversion and resolved-pipeline publication return detached data.
|
||||
- The resolved PromptKit transient-retry budget is always passed to the shared
|
||||
built-in client; document presence preserves an explicit zero.
|
||||
- The resolved digest changes when resolved composition changes and never
|
||||
includes itself.
|
||||
`Config.Resolve` clones configuration, resolves an empty programmatic cache-root
|
||||
selection, validates structure, and passes selected pipeline/lanes, profile/cache
|
||||
overrides, and exact reference changes to `pipeline.ResolvePipeline`.
|
||||
The framework resolves module/default/validator/type/reference compatibility and
|
||||
concrete producer controls before execution. Input slices/maps and the original
|
||||
configuration do not become mutable aliases of the effective result.
|
||||
|
||||
## Focused Tests
|
||||
Presence-aware selections preserve explicit invocation profile clearing, cache
|
||||
overrides, and zero semantic-turn controls rather than collapsing into omission.
|
||||
Deterministic bindings remain
|
||||
separate from LLM roles. A resolved-composition digest is not an accepted-artifact
|
||||
cache key; [state ownership](state.md) is a different boundary.
|
||||
|
||||
- **internal/core/config/document_test.go** and
|
||||
**operational_document_test.go** cover ConfigKit document loading,
|
||||
normalization, imports, overlays, and overrides.
|
||||
- **internal/core/config/validation_contract_test.go** covers configuration
|
||||
invariants and contextual failures.
|
||||
- **internal/core/config/effective_config_contract_test.go** covers defaults,
|
||||
selections, overrides, resolution context, digest changes, and ownership.
|
||||
- **internal/core/config/loaded_config_test.go** covers safe effective and
|
||||
explanation rendering; **redaction_test.go** covers resolved-pipeline
|
||||
redaction and reference-content exclusion.
|
||||
Preflight consumes resolved data, the catalog, and the captured engine when model
|
||||
roles are selected. It checks static initial/correction/validator dependencies and
|
||||
applicable explicit/default profiles. It does not load configuration, construct
|
||||
modules, read reference/input bytes, or manufacture a request to prove readiness.
|
||||
Partial profiles remain valid; dynamic request selections are runtime checks.
|
||||
|
||||
Run **go test ./internal/core/config** after changing this boundary. Changes to
|
||||
the handoff or resolved-composition semantics also need the focused framework
|
||||
pipeline tests.
|
||||
## Inspection And Sensitive Values
|
||||
|
||||
Native whole-app and focused views use retained ConfigKit evidence. Existing
|
||||
effective/explanation writers delegate to that result rather than walking YAML.
|
||||
Application sensitivity rules and native resource sensitivity participate in the
|
||||
same definition. Source/runtime option containers must not be independently
|
||||
reserialized as an alternative secret-handling mechanism.
|
||||
|
||||
Do not infer that arbitrary source, prompt, provider, or module-option strings
|
||||
are safe to disclose. Capture uses native inspection and exact marked-secret
|
||||
paths, not heuristic application-text rewriting. See [CLI composition](cli.md)
|
||||
and [LLM integration](llm.md) for those consumers.
|
||||
|
||||
## Focused Verification
|
||||
|
||||
- [Document/native settings tests](../../internal/core/config/native_settings_test.go)
|
||||
prove typed handoff, native rules, source-relative messages, and secret-aware views.
|
||||
- [Path tests](../../internal/core/config/document_paths_test.go) and
|
||||
[inheritance tests](../../internal/core/config/inheritance_test.go) cover origins,
|
||||
explicit presence, overlays, and source/runtime ownership.
|
||||
- [Control tests](../../internal/core/config/validation_policy_contract_test.go)
|
||||
cover loaded omission/zero and resolved controls.
|
||||
- [Effective resolution tests](../../internal/core/config/effective_config_contract_test.go)
|
||||
cover catalog checks, selection, and detached results.
|
||||
- [CLI loader tests](../../internal/cli/native_load_test.go) cover actual production
|
||||
underlays, strict companions, imports, frozen messages, and distinct selections.
|
||||
[Schema-load tests](../../internal/cli/native_schema_load_test.go) exercise native
|
||||
operator schema preparation/execution.
|
||||
|
||||
Use focused configuration and CLI tests when changing this handoff. The
|
||||
[development guide](../development.md#validation) owns repository-wide checks.
|
||||
|
||||
[handoff]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/consumers/pkg-promptkit.md#reuse-an-application-owned-load
|
||||
|
||||
Reference in New Issue
Block a user