Refresh native configuration, CLI, and LLM internal guides
This commit is contained in:
@@ -1,204 +1,157 @@
|
||||
# CLI Internals
|
||||
|
||||
This document describes **internal/cli**, Notarius's production composition
|
||||
root. The [CLI reference](../cli.md) owns command syntax and exit statuses;
|
||||
[Configuration](../config.md) owns configuration values; and
|
||||
[Operations](../operations.md) owns filesystem layout, recovery, and operator
|
||||
procedures.
|
||||
`internal/cli` is the production composition root. [CLI Reference](../cli.md)
|
||||
owns syntax, reports, and outcome codes; [Configuration](../config.md) owns
|
||||
settings; [Operations](../operations.md) owns physical state and recovery.
|
||||
This guide explains declaration, loading, and orchestration owners, not another
|
||||
public command manual.
|
||||
|
||||
## Inputs, Outputs, And Boundaries
|
||||
## Native Application Construction
|
||||
|
||||
The CLI accepts process arguments, standard streams, and injectable options
|
||||
used by tests and embedding code. It writes command results to the supplied
|
||||
streams and returns a process exit status. For a run, it also creates the
|
||||
production catalog and runtime collaborators, hands a prepared pipeline and
|
||||
source bytes to the framework, and places the logical files returned by the
|
||||
runner.
|
||||
[NewApplication](../../internal/cli/application.go) returns a native
|
||||
`*commandkit.App` from application identity and injectable options. Construction
|
||||
declares commands without configuration discovery, engine creation, provider
|
||||
calls, or run-storage allocation. The executable supplies build identity;
|
||||
`RunWithOptions` is a convenience execution entry point, not a manual dispatcher.
|
||||
|
||||
It is the only boundary allowed to compose concrete registries, LLM clients,
|
||||
cache/checkpoint collaborators, debug recorders, and physical output paths.
|
||||
Pipeline modules receive interfaces and request data rather than CLI streams or
|
||||
filesystem roots. The [Architecture](../policy/architecture.md) defines this
|
||||
composition-root boundary; [Pipeline Internals](pipeline.md) owns resolution,
|
||||
preparation, and runner mechanics after their inputs are supplied.
|
||||
The constructor wires:
|
||||
|
||||
## Dispatch And Configuration Handoff
|
||||
- native help/version and installed shell completion;
|
||||
- application pipeline declarations and a native run alias;
|
||||
- ckconfig whole-application and focused PromptKit scopes, sharing one loader;
|
||||
- ckprompt resource/requirements inspection, concrete request validation,
|
||||
prepared preview, namespace selection, and explicit smoke testing.
|
||||
|
||||
The root dispatcher handles help, version reporting, configuration validation,
|
||||
effective-configuration inspection, explanation inspection, pipeline listing,
|
||||
and a pipeline run. Version reporting resolves build
|
||||
information through `internal/buildinfo` before production composition, so the
|
||||
diagnostic remains available without configuration or runtime collaborators.
|
||||
The public syntax, streams, exit classes, and version semantics are defined by
|
||||
the [CLI reference](../cli.md). Other root commands normalize injectable
|
||||
options before dispatch so that a missing production dependency fails as a
|
||||
command error rather than reaching execution.
|
||||
ckconfig adapters use the original application result and focused native subtree
|
||||
view. ckprompt uses an engine loader through its standard builders rather than
|
||||
application-owned copies of inspection DTOs. Ordinary resource inspection does
|
||||
not become request preparation or an implicit probe. The
|
||||
[native CLI reference][native-cli] owns parser/report mechanics.
|
||||
|
||||
Commands that need configuration use one shared ConfigKit-backed loader. The
|
||||
CLI constructs an exclusive file source for an explicit flag or environment
|
||||
selection, or a native required search source, then passes it to `LoadSource`
|
||||
with the requested overlay and only explicit configuration-valued run overrides.
|
||||
It receives a **LoadedConfig** and uses its retained main-source path for
|
||||
invocation provenance without a second search. The
|
||||
configured discovery and precedence contract is in
|
||||
[Configuration](../config.md), while the loading and resolution mechanics are
|
||||
in [Configuration Internals](configuration.md).
|
||||
## Invocation-Local Loading
|
||||
|
||||
`config effective` and `config explain` render the loaded ConfigKit view by
|
||||
default and do not resolve pipelines or construct providers. With explicit
|
||||
`--scope promptkit`, they construct the PromptKit inspection engine from the
|
||||
same captured source groups used by execution. Effective output is PromptKit's
|
||||
redacted YAML; explanation output shows its source history. `config profile`
|
||||
uses that engine to emit one neutral Notarius-owned JSON summary for the exact
|
||||
requested profile ID. These inspection commands load no run pipeline, prepare
|
||||
no request, resolve no credential value, and make no provider call. Ordinary
|
||||
prompt and provider-parameter content is not generally secret-free even though
|
||||
credential-shaped provider parameter keys are redacted.
|
||||
[nativeLoaders](../../internal/cli/configkit_load.go) belongs to one invocation.
|
||||
It selects the main source and assembles one layered ConfigKit application
|
||||
result. Engine construction calls `promptkit.NewFromConfig` on that result,
|
||||
with runtime-only client options; it does not load YAML again.
|
||||
|
||||
Configuration validation without a selected pipeline checks structural
|
||||
configuration only. Validation with a selected pipeline also builds the
|
||||
effective catalog, resolves the pipeline, and verifies every explicit effective
|
||||
PromptKit profile. Selected LLM-backed input, chunk, lane, output, and validator
|
||||
profiles are inspected against the configured PromptKit source and backend
|
||||
registrations without preparing a request or performing generation. Engine
|
||||
construction validates effective prompt definitions and captures message content;
|
||||
an unknown or invalid profile fails before pipeline preparation.
|
||||
Credential availability remains an execution-time concern. Pipeline listing
|
||||
validates configuration before returning normalized, sorted identifiers.
|
||||
The retained selection includes supplied-value state, overlay, and override
|
||||
values. Repeated identical selections share a result/engine, while distinct
|
||||
comparison selections cannot be memoized into one side. There is no
|
||||
process-global loader cache. The selected main's native context anchors companion
|
||||
imports; source edits do not mutate an existing captured engine.
|
||||
|
||||
## Production Composition
|
||||
[Configuration Internals](configuration.md) owns definition, normalization,
|
||||
origins, and conversion. Shared loaders allocate no output/cache/debug data.
|
||||
|
||||
The production composition helper allocates every framework registry and the
|
||||
prompt-asset registry, then registers the generic, Seriatim, and D&D module
|
||||
families in that order. The resulting registries provide both the module
|
||||
catalog used for resolution and the concrete constructors used for preparation.
|
||||
Tests may provide a catalog or registries instead; production code must not
|
||||
silently merge an injected partial catalog with production registrations.
|
||||
## Catalog And Preflight Boundaries
|
||||
|
||||
The production LLM factory builds one PromptKit-backed client from the native
|
||||
source groups assembled from PromptKit catalogs, registered application prompt
|
||||
and fallback assets, the optional operator file or recursively selected
|
||||
directory, and the optional local backend registration. The same construction
|
||||
path serves runtime, preflight, and configuration inspection. It attaches the
|
||||
profile-provenance recorder, captures one immutable PromptKit configuration
|
||||
identity, creates one scheduler from the effective global LLM limit, and wraps
|
||||
the client before it reaches modules. Registration and LLM construction errors
|
||||
are returned before a pipeline is prepared. Configuration fields remain in
|
||||
[Configuration](../config.md#promptkit-profiles); source ownership, capture,
|
||||
identity, and adapter mechanics remain in [LLM Runtime](llm.md).
|
||||
[catalog.go](../../internal/cli/catalog.go) composes generic, Seriatim, and D&D
|
||||
registrars into typed registries, codecs, evidence projectors, and validator
|
||||
defaults. Registration does not build an asset/schema registry or an LLM service.
|
||||
Injected registries/catalogs are alternative sources, not implicitly augmented
|
||||
with production entries.
|
||||
|
||||
The factory also accepts `LLMRuntimeOverrides`, whose reasoning pointer
|
||||
preserves inherit, replace, and clear states across the composition boundary.
|
||||
Run orchestration constructs this value from the mutually exclusive
|
||||
`--reasoning-effort` and `--clear-reasoning-effort` controls. Absence preserves
|
||||
a nil pointer, replacement is trimmed, and clear uses a non-nil empty string.
|
||||
The same override reaches the one shared production client, checkpoint
|
||||
identity, and debug invocation metadata. Persistent reasoning configuration
|
||||
remains owned by PromptKit profiles; Notarius configuration has no reasoning
|
||||
field.
|
||||
[pipeline_commands.go](../../internal/cli/pipeline_commands.go) separates:
|
||||
|
||||
- listing: enumerate captured configured IDs without a catalog/engine;
|
||||
- showing: domain resolution and composition description, without profile expansion;
|
||||
- validating: domain resolution plus known-resource preflight, supplying the
|
||||
same captured engine only when model roles require it.
|
||||
|
||||
[Preflight](../../internal/preflight/preflight.go) accepts collaborators/data;
|
||||
it does not load configuration, register/construct modules, read execution
|
||||
inputs, render CLI reports, or allocate state. Partial profiles pass native
|
||||
inspection, with concrete request readiness deferred until preparation.
|
||||
|
||||
## Run Inputs And Native Overrides
|
||||
|
||||
[run_input.go](../../internal/cli/run_input.go) binds typed application inputs
|
||||
and repeated literal selections. [execution_overrides.go](../../internal/cli/execution_overrides.go)
|
||||
uses reusable ckprompt resource/execution inputs, preserving omission versus
|
||||
explicit empty/false/zero. Only profile/session and execution controls needed
|
||||
by pipeline runs are selected; prompt inputs stay module-owned.
|
||||
|
||||
The resolved run-wide profile selection reaches every LLM-backed role. Native
|
||||
execution overrides travel through operation requests to producers, validators,
|
||||
and continuation, without mutating loaded settings. The
|
||||
[public reference](../cli.md#run-wide-llm-overrides) owns exact spellings/semantics.
|
||||
|
||||
Reference selectors are expanded at the CLI boundary using registered target
|
||||
specifications. Pipeline/lane actions become exact binding changes after
|
||||
specificity/conflict handling. Generic resolution receives exact reference
|
||||
bindings/unbinds, not CLI selector grammar. External/generated conflicts and
|
||||
required-slot/type checks remain with domain resolution; materialization follows.
|
||||
|
||||
[session.go](../../internal/cli/session.go) resolves the effective session from
|
||||
explicit supplied input or the input-module/raw-byte policy before runner work.
|
||||
Framework/model code transports it rather than deriving another session.
|
||||
|
||||
## Run Orchestration
|
||||
|
||||
After parsing and validating a run invocation, the CLI performs this ordered
|
||||
handoff:
|
||||
[pipeline_run.go](../../internal/cli/pipeline_run.go) performs the ordered handoff:
|
||||
|
||||
1. load and validate configuration with ConfigKit command overrides;
|
||||
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||
when requested;
|
||||
3. build the effective catalog, resolve requested reference changes, resolve
|
||||
the effective pipeline, and inspect its explicit effective PromptKit
|
||||
profiles;
|
||||
4. materialize external or generated references and record redacted invocation
|
||||
and resolution provenance when debug capture is enabled;
|
||||
5. construct registries, the scheduled LLM client, and prepared modules;
|
||||
6. read the source input once, resolve its effective session from the explicit
|
||||
override or resolved input module and raw bytes, then construct requested
|
||||
checkpoint collaborators and invoke the framework runner with that same
|
||||
value; and
|
||||
7. write the runner's logical output files only after a successful run, then
|
||||
complete the command report and user-facing result.
|
||||
1. Load/validate the selected application result and derive runtime configuration.
|
||||
2. Generate a safe run identity and allocate requested capture.
|
||||
3. Capture selected configuration, compose registries, expand reference actions,
|
||||
and resolve the effective pipeline.
|
||||
4. Obtain the shared engine when needed and perform offline preflight.
|
||||
5. Materialize references, construct prepared typed modules, read input bytes,
|
||||
and resolve the effective session.
|
||||
6. Construct the accepted-result cache and optional recorder collaborator,
|
||||
create one native execution group, and invoke the runner with independent
|
||||
worker and execution selections.
|
||||
7. Report process diagnostics, publish returned logical files only after runner
|
||||
success, finalize requested capture, then submit the native completion report.
|
||||
|
||||
Preparation happens before source parsing, so module construction and
|
||||
dependency failures cannot begin stage execution. The CLI also preserves the
|
||||
framework's result and warning information when it writes summaries and the
|
||||
final command result. Detailed state lifecycle, resume handling, and physical
|
||||
path confinement are maintained in [Run State Internals](state.md) and
|
||||
[Operations](../operations.md).
|
||||
Modules receive the shared engine, operation data, and explicit collaborators,
|
||||
not physical roots or CommandKit reporters. Detailed worker, assessment,
|
||||
eligibility, and cache algorithms belong in [Pipeline Internals](pipeline.md);
|
||||
physical publication/capture collaborators belong in [Run State Internals](state.md).
|
||||
|
||||
The CLI owns the versioned generated-session policy and resolves the sole
|
||||
effective value before checkpoint construction. It records that value in the
|
||||
final debug invocation summary when capture is enabled and passes it unchanged
|
||||
to checkpoint identity and `pipeline.RunInput`. The public flag and stability
|
||||
contract are defined by the [CLI reference](../cli.md#run); framework and LLM
|
||||
packages only transport the supplied value.
|
||||
Requested capture allocates before engine/preflight and finalizes once.
|
||||
Primary operation causes and additional capture failures remain available.
|
||||
Later report delivery failure does not rerun model work, remove valid artifacts,
|
||||
or terminalize capture again.
|
||||
|
||||
The CLI also owns the scope grammar for reference flags. It enumerates the
|
||||
selected chunk and lane targets from registered module specifications, expands
|
||||
pipeline- and lane-scoped actions into exact stage-and-lane bindings, and
|
||||
resolves overlapping bind and unbind actions by specificity before calling
|
||||
configuration resolution. The generic pipeline therefore receives only exact
|
||||
`ReferenceBinding` and `ReferenceUnbind` values and has no knowledge of CLI
|
||||
selector syntax. Configuration resolution retains ownership of configured
|
||||
external/generated conflicts, required slots, and module compatibility; file
|
||||
materialization still occurs afterward.
|
||||
## Native Reporting And Outcomes
|
||||
|
||||
For `run --json`, the CLI constructs and encodes its private run-result receipt
|
||||
after a successful runner result is available, before it publishes logical
|
||||
output files. It writes the prepared receipt to standard output only after
|
||||
output publication and requested debug terminalization succeed. A receipt-write
|
||||
failure exits with runtime status 1 and may leave partial standard-output bytes,
|
||||
but the already-published output bundle remains complete and requested debug
|
||||
reporting remains successfully terminalized. The CLI reports a bounded
|
||||
command-owned error and does not repeat terminal reporting. The receipt remains
|
||||
a CLI reporting concern rather than a framework or output-module responsibility;
|
||||
its public contract is the
|
||||
[native completion report](../cli.md#completion-report).
|
||||
[run_result.go](../../internal/cli/run_result.go) submits one small application
|
||||
value with a native text template. Native serializers render that same value.
|
||||
Domain files, not a persisted report or ledger, are the deliverables.
|
||||
Fallback qualification is independent of optional diagnostic visibility.
|
||||
|
||||
## Failure Mapping And Terminal Reporting
|
||||
Warnings/advisories keep their domain classification and use the reporter's
|
||||
diagnostic channel. Outcome helpers retain native configuration/preflight causes
|
||||
and distinguish identifiable invalid input from operational/mixed failures.
|
||||
CommandKit retains rendering/output/diagnostic delivery evidence separately from
|
||||
the operation cause; quiet mode does not mask such failures.
|
||||
|
||||
Argument, flag, and invocation-combination failures are reported to standard
|
||||
error before runtime composition and use the syntax error class. Once an
|
||||
invocation is syntactically valid, configuration loading and validation,
|
||||
resolution, registration, profile checks, reference materialization, module
|
||||
construction, input reads, runner failures, output publication, and requested
|
||||
debug handling use the runtime failure class. The public status numbers and
|
||||
stream contract are defined in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
|
||||
The [completion report](../cli.md#completion-report) owns fields and
|
||||
[stream/outcome contract](../cli.md#output-streams-and-exit-statuses) owns process
|
||||
behavior. There is no universal Notarius envelope or parallel serializer.
|
||||
|
||||
When debug capture has been allocated, one command-state value records the
|
||||
known run result. Guarded terminalization writes a success report once, or
|
||||
attempts a failure report and error record once. A persistence failure is
|
||||
reported in addition to the original failure and never replaces it. If a debug
|
||||
path exists, failure output includes that path so the retained diagnostic data
|
||||
is discoverable.
|
||||
## Focused Verification
|
||||
|
||||
## Invariants To Preserve
|
||||
- [Application tests](../../internal/cli/application_test.go) cover lazy built-ins,
|
||||
shared scopes, selection evidence, and native diff outcomes.
|
||||
- [Loader tests](../../internal/cli/native_load_test.go) and
|
||||
[schema tests](../../internal/cli/native_schema_load_test.go) cover native
|
||||
composition, captured contexts, and operator contracts.
|
||||
- [Pipeline command tests](../../internal/cli/pipeline_commands_test.go) cover
|
||||
listing/showing/preflight without execution inputs or state.
|
||||
- [Override tests](../../internal/cli/execution_overrides_test.go) and
|
||||
[run tests](../../internal/cli/pipeline_run_test.go) cover supplied values,
|
||||
routing, references, sessions, and run ordering.
|
||||
- [Reference tests](../../internal/cli/reference_contract_test.go) cover selector
|
||||
specificity, generated conflicts, and materialization boundaries.
|
||||
- [Native executable tests](../../internal/cli/native_process_test.go) cover the
|
||||
real command routes and fake-provider presentations.
|
||||
- [Completion tests](../../internal/cli/run_result_command_test.go) and
|
||||
[recovery contracts](../../internal/cli/recovery_contract_test.go) cover retained
|
||||
publication, capture/delivery failures, reuse, fallback, and native facts.
|
||||
|
||||
- Only the CLI composes production implementations and physical runtime roots.
|
||||
- Configuration and resolved composition failures occur before module
|
||||
preparation or source parsing.
|
||||
- A runner's logical files are published only after a successful run.
|
||||
- Production registries and a caller-supplied catalog or registries are
|
||||
alternative composition sources, not an implicit mixture.
|
||||
- A requested debug bundle has one terminal report attempt; its persistence
|
||||
errors supplement rather than obscure the primary command error.
|
||||
- User-facing flags, paths, exit codes, and configuration fields are defined
|
||||
by their public documentation, not duplicated here.
|
||||
Use focused CLI tests and pair with configuration/preflight tests when those
|
||||
boundaries change. [Development](../development.md#validation) owns wider checks.
|
||||
|
||||
## Focused Tests
|
||||
|
||||
- **internal/cli/command_contract_test.go** covers dispatch, help, syntax and
|
||||
runtime error classes, discovery, validation, and listing.
|
||||
- **internal/cli/run_contract_test.go** covers the run handoff, publication,
|
||||
debug reporting, and command-owned state collaborators.
|
||||
- **internal/cli/production_contract_test.go** covers registrar composition,
|
||||
production catalog contents, assets, and representative configuration
|
||||
validation.
|
||||
- **internal/cli/reference_contract_test.go** covers scoped CLI reference
|
||||
expansion, specificity, bind/unbind conflicts, generated-reference
|
||||
protection, origin separation, and materialization boundaries.
|
||||
- **internal/cli/state_hardening_test.go** covers safe run identity, state
|
||||
roots, and failure ordering.
|
||||
|
||||
Run **go test ./internal/cli** after changing command composition or command
|
||||
behavior. Pair it with **go test ./internal/core/config** when the configuration
|
||||
handoff changes.
|
||||
[native-cli]: https://forge.maximumdirect.net/base-layer/commandkit/src/tag/v0.7.0/docs/cli.md
|
||||
|
||||
@@ -1,161 +1,148 @@
|
||||
# Configuration Internals
|
||||
|
||||
This document describes the maintainer-facing configuration boundary in
|
||||
**internal/core/config**. The [Configuration](../config.md) reference owns the
|
||||
file format, fields, defaults, precedence contract, and selectable keys. The
|
||||
[CLI reference](../cli.md) owns command syntax; this document does not redefine
|
||||
either interface.
|
||||
This guide explains the maintained configuration boundary and its composition-root
|
||||
handoff. [Configuration](../config.md) owns source selection, syntax, fields,
|
||||
defaults, and selectable modules; [CLI Reference](../cli.md) owns invocation.
|
||||
[Architecture](../policy/architecture.md#configuration-and-provenance) defines
|
||||
ownership. Do not infer operator contracts from private helper choices here.
|
||||
|
||||
## Boundary
|
||||
## Owners And Handoffs
|
||||
|
||||
The configuration package turns a selected ConfigKit-composed YAML document
|
||||
into a validated, independently owned configuration. It then resolves a
|
||||
requested pipeline against a module catalog before the framework prepares or
|
||||
runs anything.
|
||||
| Owner | Responsibility |
|
||||
| --- | --- |
|
||||
| [core/config definition](../../internal/core/config/load.go) | Typed application document, native PromptKit configuration rules, application resolution/validation. |
|
||||
| [CLI loaders](../../internal/cli/configkit_load.go) | Main-source selection, native underlays, invocation-local result/engine reuse. |
|
||||
| [LoadedConfig](../../internal/core/config/loaded_config.go) | Retain the original result and derive independently owned runtime pipeline configuration. |
|
||||
| [Config.Resolve](../../internal/core/config/effective_config.go) | Resolve a selected pipeline against the supplied catalog. |
|
||||
| [preflight](../../internal/preflight/preflight.go) | Check known native resources from resolved composition and supplied engine. |
|
||||
| Pipeline preparation | Construct typed execution collaborators from resolved, materialized inputs; see [Pipeline Internals](pipeline.md). |
|
||||
|
||||
| Boundary | Inputs | Outputs | Does not own |
|
||||
| --- | --- | --- | --- |
|
||||
| Loading | Selected ConfigKit source, overlay, overrides, and invocation directory | **LoadedConfig** with ConfigKit inspection metadata and a populated **Config** | Choosing the source or reporting a command result. |
|
||||
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
|
||||
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
|
||||
| Inspection | **LoadedConfig** and an optional JSON Pointer | Redacted effective YAML or explanation text | Pipeline selection, module preparation, or provider work. |
|
||||
The production flow keeps one application-owned
|
||||
`*configkit.Result[config.Document]`. That result feeds runtime conversion,
|
||||
whole-app/focused inspection, and `promptkit.NewFromConfig`. Rendered YAML is
|
||||
an operator view, not an interchange format for constructing the engine.
|
||||
|
||||
The CLI describes the selected source, invokes this package, and supplies the
|
||||
result to the framework. Configuration never reads an input file, constructs a
|
||||
module, or creates output, cache, or debug paths. Those responsibilities remain
|
||||
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
|
||||
[run-state](state.md) boundaries.
|
||||
## Definition And Application Meaning
|
||||
|
||||
## Loading And Validation
|
||||
`Document` contains native `promptkit.Settings` at its PromptKit field.
|
||||
`Definition` composes application defaults, version policy, sensitivity,
|
||||
identified step lists, resolution, and validation with `promptkit.Configure`.
|
||||
Native resource rules therefore participate in the same load rather than being
|
||||
validated by a separate application parser.
|
||||
|
||||
`Load` selects an exclusive file; `LoadSource` accepts the CLI's source and
|
||||
enables imports through the same definition and callback path. ConfigKit composes
|
||||
the main input, imports, and optional overlay, then expands native inheritance
|
||||
before typed decoding. Explicit CLI assignments follow decoding and precede
|
||||
application resolution. The document definition supplies typed
|
||||
defaults and structural validation; Notarius's recorded resolution changes
|
||||
normalize configured paths after decode. Reference paths resolve before key
|
||||
normalization moves their source coordinates. Ordinary list leaf edits are
|
||||
recorded at the leaf; a changed step identity requires a covering list record.
|
||||
Actual map-key moves retain structural records. Path normalization follows the source
|
||||
of the surviving value, including a value copied through inheritance, rather
|
||||
than the inheritance directive's location. Inherited input remains explicit
|
||||
for presence-sensitive behavior, including zero, false, and empty values;
|
||||
existing field validation still governs whether the resulting values are valid.
|
||||
ConfigKit's inspection history retains inheritance transfers alongside later
|
||||
normalization, and effective configuration renders the normalized values used
|
||||
by runtime conversion. The private typed document consumes the directive, so
|
||||
no upstream inheritance type or unresolved directive reaches runtime pipeline
|
||||
or module contracts. See [Configuration](../config.md#sibling-inheritance)
|
||||
for the operator syntax and field rules. Only
|
||||
`NOTARIUS_CONFIG` participates in discovery; retired field-level environment
|
||||
variables do not participate in loading. Version 5 decoding rejects scalar
|
||||
module bindings and scalar external references. ConfigKit retains redacted,
|
||||
historical inspection metadata while **LoadedConfig** keeps that implementation
|
||||
detail private behind effective and explanation writers. `MainSourcePath` reads
|
||||
the non-imported main file from retained sources, and runtime conversion stays detached.
|
||||
Application callbacks normalize pipeline identifiers and operational paths while
|
||||
retaining presence/source history. The per-load state tracks original coordinates
|
||||
as map keys move. Paths are resolved before normalization loses their source
|
||||
coordinates. Inherited values retain their contributing origins; CLI path
|
||||
assignments use invocation origin. See
|
||||
[document paths](../../internal/core/config/document_paths.go) and
|
||||
[pipeline document normalization](../../internal/core/config/pipeline_document.go).
|
||||
|
||||
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||
concurrency settings, malformed bindings and references, invalid retries, and
|
||||
invalid pipeline, step, or lane structure. PromptKit local-backend validation
|
||||
accepts only an absolute HTTP or HTTPS endpoint with a host and no user
|
||||
information, query, or fragment, and rejects a negative local concurrency
|
||||
limit. It also rejects a negative `promptkit.transient_retries` value; decoded
|
||||
omission resolves to two, while explicit zero remains a distinct opt-out. Its
|
||||
errors retain the closest known pipeline, lane, and binding context.
|
||||
It deliberately does not require modules to be registered: that requires a
|
||||
catalog and belongs to resolution.
|
||||
ConfigKit owns parsing, imports, overlays, inheritance, typed composition, and
|
||||
provenance. The application supplies meaning through supported rules. It does
|
||||
not implement another inheritance resolver or translate retired inputs.
|
||||
[The public reference](../config.md#imports-and-sibling-inheritance) owns exact
|
||||
presence/null/list rules and processing order.
|
||||
|
||||
The exact user-selectable values and validation rules are defined in
|
||||
[Configuration](../config.md). Keep document schema, source-aware normalization,
|
||||
runtime conversion, and that reference aligned in the same change.
|
||||
`Config.Validate` checks application-only state and pipeline structure without
|
||||
requiring registered modules. Catalog-dependent compatibility belongs to
|
||||
resolution. Configuration does not parse a source transcript, materialize
|
||||
references, construct modules, or allocate run storage.
|
||||
|
||||
## Effective Resolution
|
||||
## Production Source Assembly
|
||||
|
||||
**Config.Resolve** derives an absent extraction-worker value in an owned copy
|
||||
and validates the configuration. A programmatically supplied worker value is
|
||||
always explicit. It normalizes the requested pipeline ID, copies the selected
|
||||
profile, and passes the non-empty command-level LLM profile override, requested
|
||||
lane selection, and reference changes to the framework resolver.
|
||||
`nativeLoaders.application` selects the main source and uses
|
||||
`configkit.LoadLayered` with the application definition, native underlays,
|
||||
selected overlay, and explicit overrides. Underlays come from bundled provider
|
||||
resources, ordinary embedded application content, and main-relative operator
|
||||
companions. Native source/filesystem contexts survive the load.
|
||||
|
||||
After module and validator selection, the resolver applies the effective
|
||||
profile policy to LLM-backed bindings only: command override, binding profile,
|
||||
pipeline profile, then the prompt default. Deterministic bindings remain
|
||||
profile-free, and no second inheritance decision occurs during execution. The
|
||||
public field definitions and precedence are owned by
|
||||
[Configuration](../config.md#pipelines).
|
||||
The embedded assets package supplies content, not registration behavior.
|
||||
The operator companion uses a native main-relative context; it follows the
|
||||
selected main rather than rediscovering the working directory. Strict peers,
|
||||
optional absence, imports/destinations, and whole-record resource replacement
|
||||
remain native composition. Source grammar and precedence belong in
|
||||
[Configuration](../config.md#configuration-discovery-and-precedence).
|
||||
|
||||
The resolver retains configured `validation_policy` overrides and derives one
|
||||
detached concrete terminal policy for the chunk producer and every lane's
|
||||
extract, merge, and normalize producers. That field-by-field inheritance is
|
||||
complete before preparation, and the effective values contribute to pipeline
|
||||
and checkpoint identity; execution does not interpret configuration defaults.
|
||||
Core `Load`/`LoadSource` entry points support selected sources and the same
|
||||
definition/conversion for direct consumers and tests. They do not replace the
|
||||
production composition root's underlay assembly. Do not accidentally introduce
|
||||
a second production load by combining those paths.
|
||||
|
||||
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||
chains, checks registered module and artifact compatibility, validates module
|
||||
options, and returns the fixed ordered pipeline shape. Positive validator retry
|
||||
budgets require an LLM-backed selected validator; deterministic validators are
|
||||
rejected during resolution. Eligible LLM-backed producer specifications also
|
||||
contribute their declared correction protocol to the resolved metadata. The
|
||||
resulting **EffectiveConfig** retains the selected ID, requested selection and reference
|
||||
changes, a clone of the input configuration, and the resolved pipeline.
|
||||
Callers may therefore retain or modify their input slices and maps without
|
||||
changing the resolved result, and later consumers cannot mutate the original
|
||||
configuration through the effective value. This ownership includes the nested
|
||||
PromptKit local-backend value.
|
||||
## Result Lifetime And Runtime Conversion
|
||||
|
||||
Resolution failures stop before module construction and source parsing. They
|
||||
include an error path for an unconfigured pipeline, missing module, missing
|
||||
capability, incompatible artifact variant, invalid option, invalid reference,
|
||||
or invalid lane selection. CLI code maps these valid-invocation failures to the
|
||||
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
|
||||
`LoadedConfig.Result()` exposes the original typed result for intentional native
|
||||
handoff. `FromResult` derives pipeline runtime settings without loading sources
|
||||
or serializing settings again. `RuntimeConfig` returns a detached copy;
|
||||
the native PromptKit configuration stays with its owning result/engine rather
|
||||
than being copied into a private runtime wrapper.
|
||||
|
||||
## Resolved Identity And Redaction
|
||||
`MainSourcePath` reads the retained non-imported main-file evidence. It does not
|
||||
repeat discovery or assume the first acquired file is the main.
|
||||
|
||||
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||
after defaults, lane selection, module bindings, reference bindings, validator
|
||||
chains, selected correction protocols, effective LLM profiles, and artifact
|
||||
schema identity have been resolved. The digest excludes
|
||||
its own stored value. It identifies resolved composition rather than raw YAML
|
||||
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||
provenance before execution; cache and checkpoint identity have additional
|
||||
owners in [Run State Internals](state.md).
|
||||
Within one invocation, `nativeLoaders` retains results by complete
|
||||
`ckconfig.Selection`, including overlay and override presence/values. Captured
|
||||
override containers are copied. A repeated identical selection returns the
|
||||
same result; distinct diff sides remain distinct selections. The engine is
|
||||
constructed lazily once for its retained result. This is invocation state,
|
||||
not a process-wide persistent loader cache.
|
||||
|
||||
Configuration publication uses ConfigKit's redacted effective YAML. Resolved
|
||||
pipeline publication uses **RedactedResolvedPipelinePayload**, which copies
|
||||
bindings and nested option containers, replaces credential-shaped values with
|
||||
**[REDACTED]**, and omits materialized reference content while retaining safe
|
||||
provenance. This redaction is deliberately narrow and does not authorize
|
||||
recording arbitrary environment values or provider requests.
|
||||
`NewFromConfig` consumes the completed result, preserving native contexts and
|
||||
inspection evidence without rerunning callbacks or adding underlays. Runtime-only
|
||||
HTTP/client settings and test provider options stay at engine construction.
|
||||
[PromptKit's application-owned-load contract][handoff] owns the library API.
|
||||
|
||||
## Invariants To Preserve
|
||||
## Effective Resolution And Preflight
|
||||
|
||||
- ConfigKit composition applies defaults, imports, an optional overlay, and
|
||||
explicit overrides in one direction with retained origin metadata.
|
||||
- A configuration is structurally valid before it is resolved, and a resolved
|
||||
pipeline is compatible with the supplied catalog before preparation begins.
|
||||
- Whitespace-normalized identifiers are unique wherever they identify a
|
||||
pipeline, step, lane, worker, or reference slot.
|
||||
- Runtime conversion and resolved-pipeline publication return detached data.
|
||||
- The resolved PromptKit transient-retry budget is always passed to the shared
|
||||
built-in client; document presence preserves an explicit zero.
|
||||
- The resolved digest changes when resolved composition changes and never
|
||||
includes itself.
|
||||
`Config.Resolve` clones configuration, resolves an empty programmatic cache-root
|
||||
selection, validates structure, and passes selected pipeline/lanes, profile/cache
|
||||
overrides, and exact reference changes to `pipeline.ResolvePipeline`.
|
||||
The framework resolves module/default/validator/type/reference compatibility and
|
||||
concrete producer controls before execution. Input slices/maps and the original
|
||||
configuration do not become mutable aliases of the effective result.
|
||||
|
||||
## Focused Tests
|
||||
Presence-aware selections preserve explicit invocation profile clearing, cache
|
||||
overrides, and zero semantic-turn controls rather than collapsing into omission.
|
||||
Deterministic bindings remain
|
||||
separate from LLM roles. A resolved-composition digest is not an accepted-artifact
|
||||
cache key; [state ownership](state.md) is a different boundary.
|
||||
|
||||
- **internal/core/config/document_test.go** and
|
||||
**operational_document_test.go** cover ConfigKit document loading,
|
||||
normalization, imports, overlays, and overrides.
|
||||
- **internal/core/config/validation_contract_test.go** covers configuration
|
||||
invariants and contextual failures.
|
||||
- **internal/core/config/effective_config_contract_test.go** covers defaults,
|
||||
selections, overrides, resolution context, digest changes, and ownership.
|
||||
- **internal/core/config/loaded_config_test.go** covers safe effective and
|
||||
explanation rendering; **redaction_test.go** covers resolved-pipeline
|
||||
redaction and reference-content exclusion.
|
||||
Preflight consumes resolved data, the catalog, and the captured engine when model
|
||||
roles are selected. It checks static initial/correction/validator dependencies and
|
||||
applicable explicit/default profiles. It does not load configuration, construct
|
||||
modules, read reference/input bytes, or manufacture a request to prove readiness.
|
||||
Partial profiles remain valid; dynamic request selections are runtime checks.
|
||||
|
||||
Run **go test ./internal/core/config** after changing this boundary. Changes to
|
||||
the handoff or resolved-composition semantics also need the focused framework
|
||||
pipeline tests.
|
||||
## Inspection And Sensitive Values
|
||||
|
||||
Native whole-app and focused views use retained ConfigKit evidence. Existing
|
||||
effective/explanation writers delegate to that result rather than walking YAML.
|
||||
Application sensitivity rules and native resource sensitivity participate in the
|
||||
same definition. Source/runtime option containers must not be independently
|
||||
reserialized as an alternative secret-handling mechanism.
|
||||
|
||||
Do not infer that arbitrary source, prompt, provider, or module-option strings
|
||||
are safe to disclose. Capture uses native inspection and exact marked-secret
|
||||
paths, not heuristic application-text rewriting. See [CLI composition](cli.md)
|
||||
and [LLM integration](llm.md) for those consumers.
|
||||
|
||||
## Focused Verification
|
||||
|
||||
- [Document/native settings tests](../../internal/core/config/native_settings_test.go)
|
||||
prove typed handoff, native rules, source-relative messages, and secret-aware views.
|
||||
- [Path tests](../../internal/core/config/document_paths_test.go) and
|
||||
[inheritance tests](../../internal/core/config/inheritance_test.go) cover origins,
|
||||
explicit presence, overlays, and source/runtime ownership.
|
||||
- [Control tests](../../internal/core/config/validation_policy_contract_test.go)
|
||||
cover loaded omission/zero and resolved controls.
|
||||
- [Effective resolution tests](../../internal/core/config/effective_config_contract_test.go)
|
||||
cover catalog checks, selection, and detached results.
|
||||
- [CLI loader tests](../../internal/cli/native_load_test.go) cover actual production
|
||||
underlays, strict companions, imports, frozen messages, and distinct selections.
|
||||
[Schema-load tests](../../internal/cli/native_schema_load_test.go) exercise native
|
||||
operator schema preparation/execution.
|
||||
|
||||
Use focused configuration and CLI tests when changing this handoff. The
|
||||
[development guide](../development.md#validation) owns repository-wide checks.
|
||||
|
||||
[handoff]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/consumers/pkg-promptkit.md#reuse-an-application-owned-load
|
||||
|
||||
@@ -1,363 +1,186 @@
|
||||
# LLM Runtime Internals
|
||||
# Native LLM Integration Internals
|
||||
|
||||
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||
completion boundary. It adapts framework requests to PromptKit, bounds
|
||||
provider calls, assembles registered prompt and schema assets, records selected
|
||||
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||
credentials, and concurrency settings belong in
|
||||
[Configuration](../config.md#promptkit-profiles) and
|
||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||
Model-executing code uses native PromptKit types directly. This guide explains
|
||||
Notarius's integration points, not a private completion service or a copied
|
||||
library manual. [Architecture](../policy/architecture.md#native-execution-boundary)
|
||||
defines the boundary; [Configuration](../config.md#promptkit-profiles) owns
|
||||
resources/settings; [CLI Reference](../cli.md#run-wide-llm-overrides) owns request
|
||||
controls; [Operations](../operations.md) owns state and handling.
|
||||
|
||||
## Structured Completion Boundary
|
||||
## Construction And Ownership
|
||||
|
||||
Modules and LLM-backed validators depend only on
|
||||
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
|
||||
version, optional profile and session IDs, named input material, variables, and
|
||||
a caller-owned decode target. The successful response returns the validated raw
|
||||
structured bytes together with non-secret provider, model, profile, and token
|
||||
metadata.
|
||||
The [CLI loader](../../internal/cli/configkit_load.go) hands the original completed
|
||||
ConfigKit application result to `promptkit.NewFromConfig`. One invocation shares
|
||||
the resulting engine across initial producers, corrections, normalization, and
|
||||
LLM validators. Runtime-only HTTP/client settings and an injected native
|
||||
`promptkit.LLMClient` remain construction options, not application YAML.
|
||||
There is no provider-neutral completion conversion or second engine per role.
|
||||
|
||||
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
|
||||
selects the private response schema, and interprets the decoded result. The
|
||||
adapter does not own source evidence, artifact conversion, normalization, or
|
||||
durable schemas. Those responsibilities remain with the module and its
|
||||
[integration contract](../integrations/).
|
||||
[Configuration Internals](configuration.md) owns native definition/source assembly
|
||||
and invocation selection reuse. PromptKit owns captured resources, request
|
||||
preparation, native validation/repair, provider transport, conversation state,
|
||||
and backend admission. See the [upstream Go consumer contract][consumer] for those
|
||||
APIs rather than reproducing their mechanics here.
|
||||
|
||||
The calling module also resolves contextual entity selections and attaches any
|
||||
application identity; PromptKit and this adapter do not own entity identity.
|
||||
Application contracts remain typed production, candidate assessment, supported
|
||||
fallback, and artifact encoding. Pure domain models, codecs, source resolution,
|
||||
and deterministic transforms do not need PromptKit merely because a surrounding
|
||||
operation invokes a model.
|
||||
|
||||
`PromptKitClient` validates the request target and prompt identity, maps each
|
||||
named material to a PromptKit inline artifact while preserving its origin URI,
|
||||
passes the supplied request session through to PromptKit's direct per-run
|
||||
session field, retains the same value as the `session_id` prompt variable for
|
||||
maintained prompt compatibility, and forwards profile selection. It does not
|
||||
derive or replace session values; the CLI owns that policy. It then creates one
|
||||
frozen prepared execution, captures its caller-owned credential-redacted
|
||||
details for debug material, and executes that exact snapshot through
|
||||
PromptKit's prepared-execution boundary. The direct field
|
||||
is authoritative for provider session behavior. A session ID is a stable,
|
||||
non-secret correlation identifier and may be exposed to providers and provider
|
||||
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||||
re-encoding the decoded target. An empty optional material is represented as
|
||||
one space so its named input is retained by PromptKit.
|
||||
## Requests And Fixed Operation Inputs
|
||||
|
||||
When a request includes semantic correction material, the adapter validates and
|
||||
defensively copies it before preparation, then appends exactly two messages
|
||||
after the ordinarily rendered prompt: the prior response as an assistant
|
||||
message and the correction guidance as a user message. Requests without a
|
||||
correction do not add messages or introduce caller roles. Ordinary request
|
||||
summaries record correction byte counts and digests only; complete messages are
|
||||
available solely in an explicitly requested debug trace.
|
||||
Model modules receive the shared engine during typed preparation. A module
|
||||
prepares transcript, references, contextual identity projections, and any
|
||||
normalization candidate mapping once. It constructs a `promptkit.RunRequest`
|
||||
with its selected prompt, profile/session, native artifact inputs, and variables.
|
||||
Response interpretation consumes `*promptkit.RunResult` and produces the
|
||||
application's typed candidate.
|
||||
|
||||
The adapter leaves the ordinary rendered message prefix, named inputs,
|
||||
variables, session, profile, execution overrides, prepared-execution path, and
|
||||
PromptKit repair policy unchanged for a corrected request. It never imports a
|
||||
PromptKit message type into a module or pipeline contract. PromptKit reports
|
||||
actual structural repair count and cumulative token usage per completion; the
|
||||
pipeline's safe terminal debug record projects those values without copying
|
||||
message content.
|
||||
[Spell extraction](../../internal/modules/dnd/extract/spells/extractor.go) is a
|
||||
representative path: fixed transcript/catalog/NPC projections feed an initial
|
||||
request; native output is decoded and canonically mapped into its durable type.
|
||||
The correction builder selects a paired prompt and supplies current guidance,
|
||||
not another copy of original source/reference material.
|
||||
|
||||
Client construction may also receive a run-wide reasoning-effort override from
|
||||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||||
creates a fresh PromptKit execution override for each request: a nil pointer
|
||||
inherits the selected profile, a non-empty value replaces it, and an empty
|
||||
value clears inherited reasoning. The CLI's mutually exclusive
|
||||
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
|
||||
states. With neither flag, profile behavior remains unchanged. Because
|
||||
production constructs one shared client, the selected state applies uniformly
|
||||
to module calls, retries, and LLM-backed validators for the whole run.
|
||||
Domain identity stays deterministic. Contextual selectors and request-local
|
||||
normalization handles resolve against fixed supplied records; models do not
|
||||
acquire authority over durable IDs, citation meaning, or typed artifact shape.
|
||||
[Module Internals](modules.md) and [D&D Internals](dnd.md) own extension/domain
|
||||
recipes.
|
||||
|
||||
An empty request profile lets the prompt select its configured default. Before a
|
||||
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||||
resolved selected LLM-backed bindings and validators, including inherited
|
||||
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||||
target without preparing a request, reading credentials, admitting capacity, or
|
||||
contacting a provider. Engine construction validates effective prompt definitions
|
||||
and captures message files and static templates. A missing or invalid profile fails before
|
||||
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||||
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||||
non-secret profile identity, provider, model, selected backend ID, and
|
||||
effective reasoning values for manifest use. Entries that differ in backend or
|
||||
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||||
retain an empty backend ID, which the published JSON omits. Successful
|
||||
completion responses and recorded profile manifests identify the adapter
|
||||
provider as `promptkit`.
|
||||
Native request overrides preserve supplied values without rewriting configured
|
||||
profiles. The producer helper's request composition preserves operation-owned
|
||||
prompt/input/capture fields and existing callbacks; validators pass native
|
||||
execution options explicitly on their own requests.
|
||||
|
||||
The CLI's profile-inspection engine and production adapter use the same source
|
||||
assembly: PromptKit catalogs, one application group containing registered
|
||||
prompts and fallback profiles, the selected operator file or directory, and
|
||||
the optional final `local` backend registration. Execution, preflight, and
|
||||
inspection therefore capture the same effective definitions. PromptKit and
|
||||
ConfigKit compose recursively across groups while retaining each input's file
|
||||
context; inputs within a group compose strictly. Operator imports are enabled
|
||||
for the selected operator input, and imports retain their declaring file's
|
||||
relative resource roots. PromptKit validates final effective definitions at
|
||||
engine construction. It does not run a separate audit that rejects a malformed
|
||||
definition superseded by a later overlay. The D&D registrar contributes its
|
||||
`dnd-extraction` fallback, and maintained D&D prompts select that logical ID
|
||||
by default. A profile selecting `backend: local` can use an operator
|
||||
`backends.local` declaration or the optional Notarius registration; see
|
||||
[native source rules](../config.md#native-promptkit-sources-and-migration).
|
||||
## Native Production And Application Assessment
|
||||
|
||||
This PromptKit integration uses the old-path ConfigKit v0.5.0 types exposed by
|
||||
PromptKit v0.12.0. Application configuration uses the distinct new-path
|
||||
ConfigKit package; those named types do not cross the adapter boundary.
|
||||
[RunProducer](../../internal/framework/pipeline/producer.go) connects ordinary
|
||||
module interpretation/correction/fallback functions to the pipeline assessor.
|
||||
It uses native `Run`, or retained prepared execution on the cache path, then
|
||||
checks errors and public validation status before interpretation. Failed native
|
||||
validation can accompany nil Go error and is not a usable semantic candidate.
|
||||
|
||||
Each engine captures its definitions at construction. A later source-file edit
|
||||
does not mutate an existing engine; constructing a new engine captures the
|
||||
updated view. Before execution, the adapter computes one immutable,
|
||||
versioned `promptkit_configuration` checkpoint identity from public captured
|
||||
inspection APIs. The private digest covers captured effective configuration,
|
||||
all effective profile targets and credential-policy metadata (including
|
||||
provider parameters), and every effective prompt ID/version and captured
|
||||
content hash. The identity also includes PromptKit and the selected OpenRouter
|
||||
and Rakestrawhome catalog versions. Local-backend concurrency settings remain
|
||||
excluded because they affect scheduling; its endpoint is represented only in
|
||||
the private digest, never as checkpoint metadata. Neither the private digest
|
||||
payload, raw sources, paths, endpoints, provider parameters, nor prompt bodies
|
||||
are published. An existing engine retains its identity after files change; a
|
||||
new engine gets the identity of its newly captured definitions. This replaces
|
||||
the former live-file profile and local-endpoint fingerprints. Run manifests
|
||||
remain independent and list only profile choices actually observed during LLM
|
||||
calls.
|
||||
Every usable initial/revised candidate receives the full applicable configured
|
||||
assessment chain once, including unchanged candidates and the final permitted
|
||||
response. Native coordination owns history and additional-turn iteration.
|
||||
Its initial callback uses already collected findings instead of reassessing the
|
||||
initial result. Module templates own correction assignments; detecting modules/
|
||||
validators own actionable guidance, separately from diagnostics.
|
||||
|
||||
## Shared Provider-Call Limit
|
||||
Native execution success is not domain acceptance. Stopped/exhausted outcomes
|
||||
are control facts. Result-less failed continuation does not recover an earlier
|
||||
candidate, and an undelivered response is not assessed or accepted. Caller
|
||||
cancellation stops without fallback. Native timeout/provider/capacity failures
|
||||
remain distinguishable through context state and typed causes.
|
||||
|
||||
Production construction creates one PromptKit client and wraps it in one
|
||||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||
queued calls in FIFO order, and removes a queued call when its context is
|
||||
cancelled. It rechecks the caller context after admission and before dispatch.
|
||||
A granted permit is released exactly once on every completion path.
|
||||
Resolved producer mode chooses terminal handling. A supported permissive fallback
|
||||
must preserve intrinsic/source/downstream invariants without repeating ordinary
|
||||
assessment or claiming semantic approval. Preparation/capture/encoding failures
|
||||
are not reasons to extend model budgets. Detailed assessment/disposition and
|
||||
handoff eligibility belong in [Pipeline Internals](pipeline.md).
|
||||
|
||||
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
|
||||
lanes, pipeline retries, and LLM-backed validators share the same provider-call
|
||||
ceiling. This ceiling is independent of pipeline worker concurrency; changing
|
||||
worker counts cannot exceed the configured LLM limit. The configuration field
|
||||
and its effective default are owned by
|
||||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||
## LLM Validators
|
||||
|
||||
PromptKit applies a second, independent admission limit when the selected
|
||||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||||
so it may narrow but cannot expand the application-wide limit. Built-in
|
||||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||||
capacity policy. A positive configured local-backend limit bounds active local
|
||||
generations inside PromptKit; zero leaves that backend unlimited there.
|
||||
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||||
only by the Notarius scheduler.
|
||||
The [combat-semantics validator](../../internal/modules/dnd/validate/scenedescriptions/combat_semantics/validator.go)
|
||||
builds one native request with its own profile and scene-local input. It uses
|
||||
the same engine and run-wide native execution/session selections, then checks
|
||||
the native error/status before decoding.
|
||||
|
||||
Successful interpretation returns a domain judgment and actionable rejection
|
||||
guidance where appropriate. Execution or interpretation failure is unavailable
|
||||
assessment with retained causes, not a semantic defect or invented instructions.
|
||||
Cancellation delivers no judgment. Validators do not coordinate another semantic
|
||||
conversation or rerun themselves to extend structural repair.
|
||||
|
||||
A completed producer/continuation releases native admission before assessment,
|
||||
so a nested validator can use that same limited backend. Application workers
|
||||
and native admission remain independent; there is no second scheduler or queue.
|
||||
Capacity handling follows producer mode, not an additional application retry.
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||||
filesystems from production module families. It retains each registered root's
|
||||
filesystem context when constructing PromptKit's application source group.
|
||||
Schema files are supplied separately through the engine's schema filesystem;
|
||||
PromptKit reads and validates a selected schema while preparing that request,
|
||||
not while constructing the configuration snapshot. Prompt and fallback
|
||||
definitions are validated after native source composition, and referenced
|
||||
prompt content is captured with the engine. Checkpoint identity uses captured
|
||||
prompt content hashes rather than rescanning live files or hashing only
|
||||
content-file paths. Raw paths and bytes are never included. The framework’s
|
||||
`promptfs` helper combines
|
||||
module-selected prompt files with reusable domain fragments without making the
|
||||
framework depend on D&D content.
|
||||
[assets/notarius.yml](../../assets/notarius.yml) imports ordinary embedded
|
||||
initial/correction/validator resources with retained filesystem contexts.
|
||||
The assets package remains a content-only leaf. Prompt declarations own default
|
||||
output format, validation, and schema selections; modules own prompt meaning
|
||||
and small static resource-ID dependencies, not behavioral asset registration.
|
||||
|
||||
LLM-facing content is embedded once by the root `assets` package. Each consumer
|
||||
uses only its scoped subtree, while the module retains ownership of its prompt
|
||||
declaration, ordered manifest, private response-schema identity, and
|
||||
registration. Shared D&D fragments are selected by D&D's shared implementation;
|
||||
the detailed convention is in [D&D Module Internals](dnd.md). This physical
|
||||
arrangement and its data-only boundary are defined by
|
||||
[Architecture](../policy/architecture.md) and
|
||||
[ADR-0011](../adr/0011-centralize-llm-assets.md), rather than by this runtime
|
||||
guide.
|
||||
Each producer pair requests a complete replacement response using its shared
|
||||
schema. Enduring principles live in initial system context; the follow-up
|
||||
supplies current findings and its domain assignment. Normalization applies
|
||||
each complete proposal afresh to fixed originals, not a union of prior groups.
|
||||
|
||||
The generic registrar is the sole production registration owner for the
|
||||
semantic-reconciliation default prompt and private response schema. The
|
||||
domain-neutral reconciliation package also exposes only its mandatory protocol
|
||||
and candidate/transcript presentation files for domain prompt manifests. D&D
|
||||
registry normalizers mount those files while retaining ownership and hashing
|
||||
of their D&D system message, semantic instructions, and complete prompt
|
||||
declaration. The response schema is therefore registered once even though
|
||||
several typed normalizers select it.
|
||||
Native preparation loads selected inline/file/root schemas. Intrinsic durable
|
||||
artifact schemas remain application-owned codec contracts and are not removed
|
||||
with duplicate model-response schema machinery. Full source/root/presence rules
|
||||
belong in [Configuration](../config.md#prompt-output-contracts-and-schemas) and
|
||||
[PromptKit formats][formats].
|
||||
|
||||
Mounted prompt assets determine a module's fingerprint. The fingerprint hashes
|
||||
only the module and shared files explicitly selected by its manifest, so an
|
||||
unrelated asset does not invalidate a checkpoint. Schema loaders validate JSON,
|
||||
attach identity and digest metadata, make defensive copies, and expose
|
||||
diagnostics without raw schema bytes.
|
||||
Native structural repair and transient retries remain upstream-owned. Notarius
|
||||
does not pin omitted counts or wrap them in blanket producer/validator retries.
|
||||
Application semantic allowance is a separate resolved control, not another
|
||||
counter around native coordination.
|
||||
|
||||
Semantic-reconciliation normalizers extend this identity with the shared
|
||||
response-schema digest, framework policy version, and complete limit-policy
|
||||
digest. Their manifest metadata records the same content-free prompt, schema,
|
||||
policy, and limit identities together with domain identity and normalization
|
||||
policies. Request-local handles, source material, proposal content, and raw
|
||||
asset bytes are not checkpoint metadata.
|
||||
## Prepared Execution And Reuse
|
||||
|
||||
Private response schemas validate a model transport envelope. They are not the
|
||||
durable artifact schema and should not be documented as an external wire
|
||||
contract. Durable formats and compatibility rules remain in the
|
||||
[integration contracts](../integrations/).
|
||||
For eligible cacheable producers, the helper prepares the initial request once,
|
||||
retains its details and opaque message-content hash, and either discards the
|
||||
handle on a usable hit or executes that same handle with `RunPrepared`.
|
||||
Unused handles are discarded on exits. Continuation retains the initial key.
|
||||
|
||||
## Prompt Maintenance And Backend Caching
|
||||
Hits decode the accepted application artifact/plan without semantic reassessment,
|
||||
exchange restoration, native execution, or historical usage. Normalizers cache
|
||||
applied artifacts, not proposals. Provider-side prompt-cache hints remain native
|
||||
request metadata, distinct from accepted-result identity. State layout and
|
||||
eligibility mechanics belong in [Run State Internals](state.md), not another
|
||||
LLM cache abstraction. The [prepared execution contract][prepared] owns native
|
||||
handle lifetime and admission behavior.
|
||||
|
||||
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||||
reuse depends on identical preceding roles, rendered bytes, and cache-control
|
||||
metadata—not merely equivalent meaning. Keep reusable shared assets
|
||||
byte-identical and preserve each prompt’s declared ordering and cache controls
|
||||
when editing it.
|
||||
## Native Evidence And Sensitive Data
|
||||
|
||||
For sibling prompts that can reuse the same source material, order universal
|
||||
shared context first, request source material next, and module-specific
|
||||
suffixes last. Put a cache boundary at a reusable prefix that is useful to the
|
||||
backend. Redundant intermediate cache boundaries do not extend that reusable
|
||||
prefix and add no value.
|
||||
The runner supplies one invocation execution group. Actual initial, continuation,
|
||||
and nested validator executions enroll once; group observations preserve native
|
||||
identity, terminal coverage, and unknown usage without copied totals or a
|
||||
durable profile inventory.
|
||||
|
||||
Prompt-family owners may choose a different sequence when their inputs and
|
||||
reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC
|
||||
normalization policies are maintained in [D&D Module Internals](dnd.md#prompt-construction).
|
||||
Do not add tests that enforce prompt prose; prompt tests should verify the
|
||||
meaningful input placement and cache controls of the prompt being changed.
|
||||
[Native capture](../../internal/framework/pipeline/native_capture.go) composes
|
||||
observers/preparation callbacks rather than replacing them. Retain prepared
|
||||
details cheaply during callbacks, then write at normal boundaries. Cache paths
|
||||
reuse existing details; debug never triggers a second preparation or model call.
|
||||
Shared callback state is synchronized. Available result-less failure facts and
|
||||
unassessed undelivered responses remain distinguishable.
|
||||
|
||||
## Validation, Repair, And Retries
|
||||
|
||||
PromptKit performs prompt rendering, provider execution, and the prompt’s
|
||||
structured-output validation. The adapter reports an empty result, validation
|
||||
failure, empty structured body, or decode failure as
|
||||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||
material when they exist. Provider failures remain operational errors rather
|
||||
than output-validation failures. Apart from documented context, capacity, and
|
||||
invalid-output categories, provider error values and types do not cross the
|
||||
adapter error chain; callers receive only a credential-redacted diagnostic.
|
||||
|
||||
`LLMGenerationError` preserves the provider-neutral failure kind (`transport`,
|
||||
`provider_status`, `malformed_response`, or generic `generation`) and any HTTP
|
||||
status. When PromptKit exhausts an eligible transient-status retry budget, it
|
||||
also records that exhaustion and the attempts in that generation. Those fields
|
||||
do not include repair generations, and zero attempts means unavailable rather
|
||||
than proof that no request was sent.
|
||||
|
||||
Every execution that reaches PromptKit returns an optional request-local
|
||||
`execution` summary. It contains only terminal aggregate attempts, retry and
|
||||
repair counts, durations, outcome and failure classification, status, and
|
||||
presence-aware token usage. Preparation failures have no summary; later local
|
||||
decode failures retain the upstream terminal summary without changing it.
|
||||
|
||||
When PromptKit rejects backend admission before generation, the adapter maps
|
||||
`promptkit.ErrCapacityExceeded` to
|
||||
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||||
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||||
type as a framework contract. When supplied, the normalized selected backend
|
||||
ID appears only in that safe application-owned diagnostic context. A canceled
|
||||
caller context takes precedence. The adapter does not retry capacity failures;
|
||||
the pipeline's existing binding attempt policy sees the operational error and
|
||||
decides whether to rerun the complete operation.
|
||||
|
||||
PromptKit executes structural repair within its structured-output flow. The
|
||||
maintained production prompt manifests declare one additional repair attempt.
|
||||
When a resolved binding supplies a repair value, the adapter inspects the
|
||||
prompt, copies its complete output contract, changes only the repair limit, and
|
||||
passes that complete replacement contract to PromptKit. This preserves the
|
||||
prompt's output format, validation mode, schema, and provider structured-output
|
||||
settings.
|
||||
|
||||
A successful repair is an ordinary successful completion, not a warning. The
|
||||
adapter reports PromptKit's actual repair count and its cumulative usage
|
||||
directly, without adding the initial and corrective counts again. Debug prompt
|
||||
material records the configured complete contract; debug response material
|
||||
records the repaired response and actual validation result. If the repair
|
||||
budget is exhausted, the adapter retains the final raw bytes and debug material
|
||||
and reports `ErrInvalidStructuredOutput`. Generation failures during an initial
|
||||
or corrective call remain provider-neutral operational errors with the same
|
||||
redaction boundary.
|
||||
|
||||
Structural repair does not replace pipeline retry behavior: a binding's
|
||||
configured retry count reruns its complete stage attempt after an operational
|
||||
or structural error, module-requested retry, or actionable semantic rejection.
|
||||
An actionable module-requested retry and a validator rejection both use the
|
||||
same correction payload when the producer exposes an exact latest response;
|
||||
feedback-free module retries reconstruct the ordinary request without appended
|
||||
messages. The pipeline owns attempt lifecycle, validation chains, and retry
|
||||
diagnostics; see
|
||||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||
[binding reference](../config.md#module-bindings-and-validators).
|
||||
|
||||
## Timeout Ownership
|
||||
|
||||
The caller context remains the outer cancellation authority. PromptKit applies
|
||||
a positive effective generation timeout as an inner request deadline for each
|
||||
built-in generation attempt; an explicit zero disables only that generation
|
||||
deadline. The HTTP client timeout is a separate transport-wide cap. Notarius
|
||||
forwards the caller context and does not install another timeout wrapper around
|
||||
PromptKit.
|
||||
|
||||
The selected PromptKit profile owns generation settings. The resolved global
|
||||
`promptkit.transient_retries` configuration is passed to the shared PromptKit
|
||||
engine and controls additional built-in attempts for eligible transient HTTP
|
||||
failures in one generation. It defaults to two; explicit zero disables those
|
||||
attempts. Notarius binding retries remain outside the adapter and repeat the
|
||||
complete module operation and validation chain.
|
||||
Operator-facing behavior is summarized in
|
||||
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||||
contract is identified in
|
||||
[PromptKit Integration](../integrations/pkg-promptkit.md).
|
||||
|
||||
## Observability And Redaction
|
||||
|
||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||
wrapper records prepared prompt and response material, timing, selected profile
|
||||
and backend, effective model parameters, and call identifiers in the run’s
|
||||
debug bundle, including material available from a failed structured completion.
|
||||
The profile-inspection CLI projects copied values into its Notarius-owned
|
||||
neutral JSON DTO; effective configuration and explanation remain PromptKit's
|
||||
redacted YAML and source-history views. Credential names and `APIKeyRequired`
|
||||
metadata are configuration metadata, not credential-availability checks.
|
||||
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||||
include `backend_id`. For a successful completion, a debug-write failure is
|
||||
surfaced; when the completion already failed, its call error remains the
|
||||
result. Debug-bundle location, retention, and handling are operational concerns
|
||||
documented in [Operations](../operations.md#debug-bundles).
|
||||
|
||||
The attempt-terminal summary is a separate safe trace record: it contains
|
||||
attempt kinds, validator outcome counts and reason codes, effective policy,
|
||||
terminal action, and repair/usage references. It excludes raw assistant
|
||||
responses and correction text. Those values can appear only in the explicitly
|
||||
requested detailed prompt and response artifacts, which require sensitive-data
|
||||
handling.
|
||||
|
||||
Run manifests receive selected profile summaries, including optional effective
|
||||
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||
source, reference, or response content. The published field semantics belong
|
||||
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||||
Provider error text is wrapped with prompt context and bearer credentials are
|
||||
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||||
available to other runtime collaborators; it does not make prompt or response
|
||||
contents safe for general logging.
|
||||
|
||||
Generation failures expose an application-owned category and optional HTTP
|
||||
status. Provider code, type, and message remain debug-only, after redaction.
|
||||
|
||||
## Failure Boundaries
|
||||
|
||||
- Construction fails for missing asset registries, mutually exclusive profile
|
||||
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||
context cancellation propagate to the calling stage with context.
|
||||
- Backend admission exhaustion is a provider-neutral operational error and is
|
||||
not classified as invalid structured output or validator rejection.
|
||||
- Malformed or schema-invalid provider output is classified separately as
|
||||
invalid structured output so the module or pipeline can apply its own retry
|
||||
and rejection policy.
|
||||
- Domain semantic checks, evidence decisions, and deterministic normalization
|
||||
run outside the provider adapter.
|
||||
[Run capture](../../internal/cli/run_capture.go) uses secret-aware configuration
|
||||
views and exact marked-secret omissions, while native snapshots preserve credential
|
||||
omissions. Do not heuristically scrub arbitrary source/prompt/response strings.
|
||||
Selected full evidence can be sensitive; private debug representations are not
|
||||
a consumer serialization contract. Recording failures reach application boundaries
|
||||
without retrying model work. [Operations](../operations.md#debug-bundles) owns
|
||||
handling and [State Internals](state.md) owns recorder lifecycle.
|
||||
|
||||
## Focused Verification
|
||||
|
||||
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
|
||||
tests when changing this boundary. Prompt changes also require the owning
|
||||
module’s asset tests, and retry or debug changes require focused pipeline or
|
||||
CLI coverage. The focused runtime and D&D checks are:
|
||||
Use real kits with a fake native provider or local fake HTTP server, not a
|
||||
replacement completion interface. Relevant owners include:
|
||||
|
||||
~~~sh
|
||||
go test ./internal/framework/llm/... ./internal/modules/dnd/...
|
||||
~~~
|
||||
- [Native loading](../../internal/cli/native_load_test.go) and
|
||||
[schema overrides](../../internal/cli/native_schema_load_test.go).
|
||||
- [Producer/continuation tests](../../internal/framework/pipeline/producer_test.go),
|
||||
[ordered assessment](../../internal/framework/pipeline/candidate_assessment_test.go),
|
||||
and [nested native validation](../../internal/framework/pipeline/native_assessment_test.go).
|
||||
- [Accepted-result reuse](../../internal/framework/pipeline/producer_cache_test.go)
|
||||
and [native capture](../../internal/framework/pipeline/native_capture_test.go).
|
||||
- [Representative extraction](../../internal/modules/dnd/extract/spells/extractor_test.go)
|
||||
and [validator execution](../../internal/modules/dnd/validate/scenedescriptions/combat_semantics/native_execution_test.go).
|
||||
- [Assembled recovery](../../internal/cli/recovery_contract_test.go) and
|
||||
[real executable checks](../../internal/cli/native_process_test.go).
|
||||
|
||||
These verify native status/causes, complete assessment, fixed inputs/history,
|
||||
override propagation, cancellation/admission, reuse, and evidence. They do not
|
||||
claim production model quality. Wider checks belong in
|
||||
[Development](../development.md#validation).
|
||||
|
||||
[consumer]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/consumers/pkg-promptkit.md
|
||||
[formats]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/formats.md
|
||||
[prepared]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.24.0/docs/consumers/pkg-promptkit.md#prepare-now-and-execute-the-same-snapshot-later
|
||||
|
||||
Reference in New Issue
Block a user