Refresh native configuration, CLI, and LLM internal guides

This commit is contained in:
2026-11-22 06:41:22 +00:00
parent 2c27695df9
commit 5db09d0203
4 changed files with 428 additions and 642 deletions

View File

@@ -1,161 +1,148 @@
# Configuration Internals
This document describes the maintainer-facing configuration boundary in
**internal/core/config**. The [Configuration](../config.md) reference owns the
file format, fields, defaults, precedence contract, and selectable keys. The
[CLI reference](../cli.md) owns command syntax; this document does not redefine
either interface.
This guide explains the maintained configuration boundary and its composition-root
handoff. [Configuration](../config.md) owns source selection, syntax, fields,
defaults, and selectable modules; [CLI Reference](../cli.md) owns invocation.
[Architecture](../policy/architecture.md#configuration-and-provenance) defines
ownership. Do not infer operator contracts from private helper choices here.
## Boundary
## Owners And Handoffs
The configuration package turns a selected ConfigKit-composed YAML document
into a validated, independently owned configuration. It then resolves a
requested pipeline against a module catalog before the framework prepares or
runs anything.
| Owner | Responsibility |
| --- | --- |
| [core/config definition](../../internal/core/config/load.go) | Typed application document, native PromptKit configuration rules, application resolution/validation. |
| [CLI loaders](../../internal/cli/configkit_load.go) | Main-source selection, native underlays, invocation-local result/engine reuse. |
| [LoadedConfig](../../internal/core/config/loaded_config.go) | Retain the original result and derive independently owned runtime pipeline configuration. |
| [Config.Resolve](../../internal/core/config/effective_config.go) | Resolve a selected pipeline against the supplied catalog. |
| [preflight](../../internal/preflight/preflight.go) | Check known native resources from resolved composition and supplied engine. |
| Pipeline preparation | Construct typed execution collaborators from resolved, materialized inputs; see [Pipeline Internals](pipeline.md). |
| Boundary | Inputs | Outputs | Does not own |
| --- | --- | --- | --- |
| Loading | Selected ConfigKit source, overlay, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the source or reporting a command result. |
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
| Inspection | **LoadedConfig** and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. |
The production flow keeps one application-owned
`*configkit.Result[config.Document]`. That result feeds runtime conversion,
whole-app/focused inspection, and `promptkit.NewFromConfig`. Rendered YAML is
an operator view, not an interchange format for constructing the engine.
The CLI describes the selected source, invokes this package, and supplies the
result to the framework. Configuration never reads an input file, constructs a
module, or creates output, cache, or debug paths. Those responsibilities remain
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
[run-state](state.md) boundaries.
## Definition And Application Meaning
## Loading And Validation
`Document` contains native `promptkit.Settings` at its PromptKit field.
`Definition` composes application defaults, version policy, sensitivity,
identified step lists, resolution, and validation with `promptkit.Configure`.
Native resource rules therefore participate in the same load rather than being
validated by a separate application parser.
`Load` selects an exclusive file; `LoadSource` accepts the CLI's source and
enables imports through the same definition and callback path. ConfigKit composes
the main input, imports, and optional overlay, then expands native inheritance
before typed decoding. Explicit CLI assignments follow decoding and precede
application resolution. The document definition supplies typed
defaults and structural validation; Notarius's recorded resolution changes
normalize configured paths after decode. Reference paths resolve before key
normalization moves their source coordinates. Ordinary list leaf edits are
recorded at the leaf; a changed step identity requires a covering list record.
Actual map-key moves retain structural records. Path normalization follows the source
of the surviving value, including a value copied through inheritance, rather
than the inheritance directive's location. Inherited input remains explicit
for presence-sensitive behavior, including zero, false, and empty values;
existing field validation still governs whether the resulting values are valid.
ConfigKit's inspection history retains inheritance transfers alongside later
normalization, and effective configuration renders the normalized values used
by runtime conversion. The private typed document consumes the directive, so
no upstream inheritance type or unresolved directive reaches runtime pipeline
or module contracts. See [Configuration](../config.md#sibling-inheritance)
for the operator syntax and field rules. Only
`NOTARIUS_CONFIG` participates in discovery; retired field-level environment
variables do not participate in loading. Version 5 decoding rejects scalar
module bindings and scalar external references. ConfigKit retains redacted,
historical inspection metadata while **LoadedConfig** keeps that implementation
detail private behind effective and explanation writers. `MainSourcePath` reads
the non-imported main file from retained sources, and runtime conversion stays detached.
Application callbacks normalize pipeline identifiers and operational paths while
retaining presence/source history. The per-load state tracks original coordinates
as map keys move. Paths are resolved before normalization loses their source
coordinates. Inherited values retain their contributing origins; CLI path
assignments use invocation origin. See
[document paths](../../internal/core/config/document_paths.go) and
[pipeline document normalization](../../internal/core/config/pipeline_document.go).
**Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. PromptKit local-backend validation
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
information, query, or fragment, and rejects a negative local concurrency
limit. It also rejects a negative `promptkit.transient_retries` value; decoded
omission resolves to two, while explicit zero remains a distinct opt-out. Its
errors retain the closest known pipeline, lane, and binding context.
It deliberately does not require modules to be registered: that requires a
catalog and belongs to resolution.
ConfigKit owns parsing, imports, overlays, inheritance, typed composition, and
provenance. The application supplies meaning through supported rules. It does
not implement another inheritance resolver or translate retired inputs.
[The public reference](../config.md#imports-and-sibling-inheritance) owns exact
presence/null/list rules and processing order.
The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep document schema, source-aware normalization,
runtime conversion, and that reference aligned in the same change.
`Config.Validate` checks application-only state and pipeline structure without
requiring registered modules. Catalog-dependent compatibility belongs to
resolution. Configuration does not parse a source transcript, materialize
references, construct modules, or allocate run storage.
## Effective Resolution
## Production Source Assembly
**Config.Resolve** derives an absent extraction-worker value in an owned copy
and validates the configuration. A programmatically supplied worker value is
always explicit. It normalizes the requested pipeline ID, copies the selected
profile, and passes the non-empty command-level LLM profile override, requested
lane selection, and reference changes to the framework resolver.
`nativeLoaders.application` selects the main source and uses
`configkit.LoadLayered` with the application definition, native underlays,
selected overlay, and explicit overrides. Underlays come from bundled provider
resources, ordinary embedded application content, and main-relative operator
companions. Native source/filesystem contexts survive the load.
After module and validator selection, the resolver applies the effective
profile policy to LLM-backed bindings only: command override, binding profile,
pipeline profile, then the prompt default. Deterministic bindings remain
profile-free, and no second inheritance decision occurs during execution. The
public field definitions and precedence are owned by
[Configuration](../config.md#pipelines).
The embedded assets package supplies content, not registration behavior.
The operator companion uses a native main-relative context; it follows the
selected main rather than rediscovering the working directory. Strict peers,
optional absence, imports/destinations, and whole-record resource replacement
remain native composition. Source grammar and precedence belong in
[Configuration](../config.md#configuration-discovery-and-precedence).
The resolver retains configured `validation_policy` overrides and derives one
detached concrete terminal policy for the chunk producer and every lane's
extract, merge, and normalize producers. That field-by-field inheritance is
complete before preparation, and the effective values contribute to pipeline
and checkpoint identity; execution does not interpret configuration defaults.
Core `Load`/`LoadSource` entry points support selected sources and the same
definition/conversion for direct consumers and tests. They do not replace the
production composition root's underlay assembly. Do not accidentally introduce
a second production load by combining those paths.
The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module
options, and returns the fixed ordered pipeline shape. Positive validator retry
budgets require an LLM-backed selected validator; deterministic validators are
rejected during resolution. Eligible LLM-backed producer specifications also
contribute their declared correction protocol to the resolved metadata. The
resulting **EffectiveConfig** retains the selected ID, requested selection and reference
changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value. This ownership includes the nested
PromptKit local-backend value.
## Result Lifetime And Runtime Conversion
Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing
capability, incompatible artifact variant, invalid option, invalid reference,
or invalid lane selection. CLI code maps these valid-invocation failures to the
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
`LoadedConfig.Result()` exposes the original typed result for intentional native
handoff. `FromResult` derives pipeline runtime settings without loading sources
or serializing settings again. `RuntimeConfig` returns a detached copy;
the native PromptKit configuration stays with its owning result/engine rather
than being copied into a private runtime wrapper.
## Resolved Identity And Redaction
`MainSourcePath` reads the retained non-imported main-file evidence. It does not
repeat discovery or assume the first acquired file is the main.
The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator
chains, selected correction protocols, effective LLM profiles, and artifact
schema identity have been resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional
owners in [Run State Internals](state.md).
Within one invocation, `nativeLoaders` retains results by complete
`ckconfig.Selection`, including overlay and override presence/values. Captured
override containers are copied. A repeated identical selection returns the
same result; distinct diff sides remain distinct selections. The engine is
constructed lazily once for its retained result. This is invocation state,
not a process-wide persistent loader cache.
Configuration publication uses ConfigKit's redacted effective YAML. Resolved
pipeline publication uses **RedactedResolvedPipelinePayload**, which copies
bindings and nested option containers, replaces credential-shaped values with
**[REDACTED]**, and omits materialized reference content while retaining safe
provenance. This redaction is deliberately narrow and does not authorize
recording arbitrary environment values or provider requests.
`NewFromConfig` consumes the completed result, preserving native contexts and
inspection evidence without rerunning callbacks or adding underlays. Runtime-only
HTTP/client settings and test provider options stay at engine construction.
[PromptKit's application-owned-load contract][handoff] owns the library API.
## Invariants To Preserve
## Effective Resolution And Preflight
- ConfigKit composition applies defaults, imports, an optional overlay, and
explicit overrides in one direction with retained origin metadata.
- A configuration is structurally valid before it is resolved, and a resolved
pipeline is compatible with the supplied catalog before preparation begins.
- Whitespace-normalized identifiers are unique wherever they identify a
pipeline, step, lane, worker, or reference slot.
- Runtime conversion and resolved-pipeline publication return detached data.
- The resolved PromptKit transient-retry budget is always passed to the shared
built-in client; document presence preserves an explicit zero.
- The resolved digest changes when resolved composition changes and never
includes itself.
`Config.Resolve` clones configuration, resolves an empty programmatic cache-root
selection, validates structure, and passes selected pipeline/lanes, profile/cache
overrides, and exact reference changes to `pipeline.ResolvePipeline`.
The framework resolves module/default/validator/type/reference compatibility and
concrete producer controls before execution. Input slices/maps and the original
configuration do not become mutable aliases of the effective result.
## Focused Tests
Presence-aware selections preserve explicit invocation profile clearing, cache
overrides, and zero semantic-turn controls rather than collapsing into omission.
Deterministic bindings remain
separate from LLM roles. A resolved-composition digest is not an accepted-artifact
cache key; [state ownership](state.md) is a different boundary.
- **internal/core/config/document_test.go** and
**operational_document_test.go** cover ConfigKit document loading,
normalization, imports, overlays, and overrides.
- **internal/core/config/validation_contract_test.go** covers configuration
invariants and contextual failures.
- **internal/core/config/effective_config_contract_test.go** covers defaults,
selections, overrides, resolution context, digest changes, and ownership.
- **internal/core/config/loaded_config_test.go** covers safe effective and
explanation rendering; **redaction_test.go** covers resolved-pipeline
redaction and reference-content exclusion.
Preflight consumes resolved data, the catalog, and the captured engine when model
roles are selected. It checks static initial/correction/validator dependencies and
applicable explicit/default profiles. It does not load configuration, construct
modules, read reference/input bytes, or manufacture a request to prove readiness.
Partial profiles remain valid; dynamic request selections are runtime checks.
Run **go test ./internal/core/config** after changing this boundary. Changes to
the handoff or resolved-composition semantics also need the focused framework
pipeline tests.
## Inspection And Sensitive Values
Native whole-app and focused views use retained ConfigKit evidence. Existing
effective/explanation writers delegate to that result rather than walking YAML.
Application sensitivity rules and native resource sensitivity participate in the
same definition. Source/runtime option containers must not be independently
reserialized as an alternative secret-handling mechanism.
Do not infer that arbitrary source, prompt, provider, or module-option strings
are safe to disclose. Capture uses native inspection and exact marked-secret
paths, not heuristic application-text rewriting. See [CLI composition](cli.md)
and [LLM integration](llm.md) for those consumers.
## Focused Verification
- [Document/native settings tests](../../internal/core/config/native_settings_test.go)
prove typed handoff, native rules, source-relative messages, and secret-aware views.
- [Path tests](../../internal/core/config/document_paths_test.go) and
[inheritance tests](../../internal/core/config/inheritance_test.go) cover origins,
explicit presence, overlays, and source/runtime ownership.
- [Control tests](../../internal/core/config/validation_policy_contract_test.go)
cover loaded omission/zero and resolved controls.
- [Effective resolution tests](../../internal/core/config/effective_config_contract_test.go)
cover catalog checks, selection, and detached results.
- [CLI loader tests](../../internal/cli/native_load_test.go) cover actual production
underlays, strict companions, imports, frozen messages, and distinct selections.
[Schema-load tests](../../internal/cli/native_schema_load_test.go) exercise native
operator schema preparation/execution.
Use focused configuration and CLI tests when changing this handoff. The
[development guide](../development.md#validation) owns repository-wide checks.
[handoff]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/consumers/pkg-promptkit.md#reuse-an-application-owned-load