notarius/docs/internal/llm.md

# LLM Runtime Internals

`internal/framework/llm` is Notarius’s provider-independent structured
completion boundary. It adapts framework requests to PromptKit, bounds
provider calls, assembles registered prompt and schema assets, records selected
profiles, and redacts provider errors. The architectural boundary is defined in
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
credentials, and concurrency settings belong in
[Configuration](../config.md#promptkit-profiles) and
[Configuration](../config.md#concurrency-output-cache-and-debug).

## Structured Completion Boundary

Modules and LLM-backed validators depend only on
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
version, optional profile and session IDs, named input material, variables, and
a caller-owned decode target. The successful response returns the validated raw
structured bytes together with non-secret provider, model, profile, and token
metadata.

The caller owns the domain behavior: it chooses the prompt, prepares inputs,
selects the private response schema, and interprets the decoded result. The
adapter does not own source evidence, artifact conversion, normalization, or
durable schemas. Those responsibilities remain with the module and its
[integration contract](../integrations/).

The calling module also resolves contextual entity selections and attaches any
application identity; PromptKit and this adapter do not own entity identity.

`PromptKitClient` validates the request target and prompt identity, maps each
named material to a PromptKit inline artifact while preserving its origin URI,
passes the supplied request session through to PromptKit's direct per-run
session field, retains the same value as the `session_id` prompt variable for
maintained prompt compatibility, and forwards profile selection. It does not
derive or replace session values; the CLI owns that policy. It then creates one
frozen prepared execution, captures its caller-owned credential-redacted
details for debug material, and executes that exact snapshot through
PromptKit's prepared-execution boundary. The direct field
is authoritative for provider session behavior. A session ID is a stable,
non-secret correlation identifier and may be exposed to providers and provider
observability. The adapter returns PromptKit’s validated raw bytes rather than
re-encoding the decoded target. An empty optional material is represented as
one space so its named input is retained by PromptKit.

When a request includes semantic correction material, the adapter validates and
defensively copies it before preparation, then appends exactly two messages
after the ordinarily rendered prompt: the prior response as an assistant
message and the correction guidance as a user message. Requests without a
correction do not add messages or introduce caller roles. Ordinary request
summaries record correction byte counts and digests only; complete messages are
available solely in an explicitly requested debug trace.

The adapter leaves the ordinary rendered message prefix, named inputs,
variables, session, profile, execution overrides, prepared-execution path, and
PromptKit repair policy unchanged for a corrected request. It never imports a
PromptKit message type into a module or pipeline contract. PromptKit reports
actual structural repair count and cumulative token usage per completion; the
pipeline's safe terminal debug record projects those values without copying
message content.

Client construction may also receive a run-wide reasoning-effort override from
the CLI factory boundary. The adapter copies the caller-owned pointer and
creates a fresh PromptKit execution override for each request: a nil pointer
inherits the selected profile, a non-empty value replaces it, and an empty
value clears inherited reasoning. The CLI's mutually exclusive
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.

An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicit profile on the
resolved selected LLM-backed bindings and validators, including inherited
pipeline profiles. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.

The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file,
the optional registered fallback profile assets, and the optional conventional
`local` backend. Preflight therefore resolves the same profile sources and
backend membership as runtime without performing generation. Fallback assets
are mounted only when at least one source is registered. The production D&D
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
prompts select that logical ID by default. PromptKit owns source precedence and
profile parsing and inheritance: an operator-provided matching profile takes precedence over a
fallback profile without Notarius merging either document.
When the registration is absent, a profile selecting `backend: local` fails
inspection instead of falling back to a built-in or endpoint-only target.

Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
identity of PromptKit's compiled-in profile catalog with a deterministic digest
of every YAML profile in the configured profile directory, or of the configured
profile file, and a deterministic digest of the flattened fallback profile
assets. The fingerprint contains neither profile content nor source paths. It
covers inherited pipeline profiles, explicit binding profiles, and
prompt-selected defaults, so changing a model or other profile setting cannot
reuse checkpoints created under the
prior profile source. This cache identity is independent of durable
profile provenance: run manifests continue to list only profiles actually
observed during LLM calls. When the local backend is registered, a second
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
stored in checkpoint identity, and the local concurrency limit is excluded
because it changes scheduling rather than execution semantics.

## Shared Provider-Call Limit

Production construction creates one PromptKit client and wraps it in one
scheduled client. The scheduler has a fixed, positive permit limit, serves
queued calls in FIFO order, and removes a queued call when its context is
cancelled. It rechecks the caller context after admission and before dispatch.
A granted permit is released exactly once on every completion path.

The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
lanes, pipeline retries, and LLM-backed validators share the same provider-call
ceiling. This ceiling is independent of pipeline worker concurrency; changing
worker counts cannot exceed the configured LLM limit. The configuration field
and its effective default are owned by
[Configuration](../config.md#concurrency-output-cache-and-debug).

PromptKit applies a second, independent admission limit when the selected
profile names a limited backend. It sits beneath the Notarius scheduled client,
so it may narrow but cannot expand the application-wide limit. Built-in
OpenRouter profiles select PromptKit's reserved backend and its upstream
capacity policy. A positive configured local-backend limit bounds active local
generations inside PromptKit; zero leaves that backend unlimited there.
Endpoint-only profiles do not select a PromptKit backend and remain limited
only by the Notarius scheduler.

## Prompt And Schema Assets

An `AssetRegistry` collects prompt, schema, and optional fallback-profile
filesystems from production module families. It flattens registered roots into
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
assets, duplicate paths, and missing prompt or schema files during preparation.
Fallback assets receive a safe content digest for checkpoint identity; raw
paths and bytes are never included. The framework’s `promptfs` helper combines
module-selected prompt files with reusable domain fragments without making the
framework depend on D&D content.

LLM-facing content is embedded once by the root `assets` package. Each consumer
uses only its scoped subtree, while the module retains ownership of its prompt
declaration, ordered manifest, private response-schema identity, and
registration. Shared D&D fragments are selected by D&D's shared implementation;
the detailed convention is in [D&D Module Internals](dnd.md). This physical
arrangement and its data-only boundary are defined by
[Architecture](../policy/architecture.md) and
[ADR-0011](../adr/0011-centralize-llm-assets.md), rather than by this runtime
guide.

The generic registrar is the sole production registration owner for the
semantic-reconciliation default prompt and private response schema. The
domain-neutral reconciliation package also exposes only its mandatory protocol
and candidate/transcript presentation files for domain prompt manifests. D&D
registry normalizers mount those files while retaining ownership and hashing
of their D&D system message, semantic instructions, and complete prompt
declaration. The response schema is therefore registered once even though
several typed normalizers select it.

Mounted prompt assets determine a module's fingerprint. The fingerprint hashes
only the module and shared files explicitly selected by its manifest, so an
unrelated asset does not invalidate a checkpoint. Schema loaders validate JSON,
attach identity and digest metadata, make defensive copies, and expose
diagnostics without raw schema bytes.

Semantic-reconciliation normalizers extend this identity with the shared
response-schema digest, framework policy version, and complete limit-policy
digest. Their manifest metadata records the same content-free prompt, schema,
policy, and limit identities together with domain identity and normalization
policies. Request-local handles, source material, proposal content, and raw
asset bytes are not checkpoint metadata.

Private response schemas validate a model transport envelope. They are not the
durable artifact schema and should not be documented as an external wire
contract. Durable formats and compatibility rules remain in the
[integration contracts](../integrations/).

## Prompt Maintenance And Backend Caching

Prompt message order and shared asset bytes are runtime behavior. Backend cache
reuse depends on identical preceding roles, rendered bytes, and cache-control
metadata—not merely equivalent meaning. Keep reusable shared assets
byte-identical and preserve each prompt’s declared ordering and cache controls
when editing it.

For sibling prompts that can reuse the same source material, order universal
shared context first, request source material next, and module-specific
suffixes last. Put a cache boundary at a reusable prefix that is useful to the
backend. Redundant intermediate cache boundaries do not extend that reusable
prefix and add no value.

Prompt-family owners may choose a different sequence when their inputs and
reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC
normalization policies are maintained in [D&D Module Internals](dnd.md#prompt-construction).
Do not add tests that enforce prompt prose; prompt tests should verify the
meaningful input placement and cache controls of the prompt being changed.

## Validation, Repair, And Retries

PromptKit performs prompt rendering, provider execution, and the prompt’s
structured-output validation. The adapter reports an empty result, validation
failure, empty structured body, or decode failure as
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
material when they exist. Provider failures remain operational errors rather
than output-validation failures. Apart from documented context, capacity, and
invalid-output categories, provider error values and types do not cross the
adapter error chain; callers receive only a credential-redacted diagnostic.

`LLMGenerationError` preserves the provider-neutral failure kind (`transport`,
`provider_status`, `malformed_response`, or generic `generation`) and any HTTP
status. When PromptKit exhausts an eligible transient-status retry budget, it
also records that exhaustion and the attempts in that generation. Those fields
do not include repair generations, and zero attempts means unavailable rather
than proof that no request was sent.

Every execution that reaches PromptKit returns an optional request-local
`execution` summary. It contains only terminal aggregate attempts, retry and
repair counts, durations, outcome and failure classification, status, and
presence-aware token usage. Preparation failures have no summary; later local
decode failures retain the upstream terminal summary without changing it.

When PromptKit rejects backend admission before generation, the adapter maps
`promptkit.ErrCapacityExceeded` to
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
type as a framework contract. When supplied, the normalized selected backend
ID appears only in that safe application-owned diagnostic context. A canceled
caller context takes precedence. The adapter does not retry capacity failures;
the pipeline's existing binding attempt policy sees the operational error and
decides whether to rerun the complete operation.

PromptKit executes structural repair within its structured-output flow. The
maintained production prompt manifests declare one additional repair attempt.
When a resolved binding supplies a repair value, the adapter inspects the
prompt, copies its complete output contract, changes only the repair limit, and
passes that complete replacement contract to PromptKit. This preserves the
prompt's output format, validation mode, schema, and provider structured-output
settings.

A successful repair is an ordinary successful completion, not a warning. The
adapter reports PromptKit's actual repair count and its cumulative usage
directly, without adding the initial and corrective counts again. Debug prompt
material records the configured complete contract; debug response material
records the repaired response and actual validation result. If the repair
budget is exhausted, the adapter retains the final raw bytes and debug material
and reports `ErrInvalidStructuredOutput`. Generation failures during an initial
or corrective call remain provider-neutral operational errors with the same
redaction boundary.

Structural repair does not replace pipeline retry behavior: a binding's
configured retry count reruns its complete stage attempt after an operational
or structural error, module-requested retry, or actionable semantic rejection.
An actionable module-requested retry and a validator rejection both use the
same correction payload when the producer exposes an exact latest response;
feedback-free module retries reconstruct the ordinary request without appended
messages. The pipeline owns attempt lifecycle, validation chains, and retry
diagnostics; see
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
[binding reference](../config.md#module-bindings-and-validators).

## Timeout Ownership

The caller context remains the outer cancellation authority. PromptKit applies
a positive effective generation timeout as an inner request deadline for each
built-in generation attempt; an explicit zero disables only that generation
deadline. The HTTP client timeout is a separate transport-wide cap. Notarius
forwards the caller context and does not install another timeout wrapper around
PromptKit.

The selected PromptKit profile owns generation settings. The resolved global
`promptkit.transient_retries` configuration is passed to the shared PromptKit
engine and controls additional built-in attempts for eligible transient HTTP
failures in one generation. It defaults to two; explicit zero disables those
attempts. Notarius binding retries remain outside the adapter and repeat the
complete module operation and validation chain.
Operator-facing behavior is summarized in
[Operations](../operations.md#operational-limits), and the pinned upstream
contract is identified in
[PromptKit Integration](../integrations/pkg-promptkit.md).

## Observability And Redaction

When debug recording is enabled, the pipeline decorates the shared client. The
wrapper records prepared prompt and response material, timing, selected profile
and backend, effective model parameters, and call identifiers in the run’s
debug bundle, including material available from a failed structured completion.
Effective parameters use PromptKit's stable lower-case JSON field names and may
include `backend_id`. For a successful completion, a debug-write failure is
surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).

The attempt-terminal summary is a separate safe trace record: it contains
attempt kinds, validator outcome counts and reason codes, effective policy,
terminal action, and repair/usage references. It excludes raw assistant
responses and correction text. Those values can appear only in the explicitly
requested detailed prompt and response artifacts, which require sensitive-data
handling.

Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong
to the [JSON output contract](../integrations/json-output.md#manifestjson).
Provider error text is wrapped with prompt context and bearer credentials are
redacted before it crosses the runtime boundary. Known-secret redaction is
available to other runtime collaborators; it does not make prompt or response
contents safe for general logging.

Generation failures expose an application-owned category and optional HTTP
status. Provider code, type, and message remain debug-only, after redaction.

## Failure Boundaries

- Construction fails for missing asset registries, mutually exclusive profile
  sources, invalid asset registration, or a non-positive scheduler limit.
- Preparation failures, unavailable explicit profiles, provider failures, and
  context cancellation propagate to the calling stage with context.
- Backend admission exhaustion is a provider-neutral operational error and is
  not classified as invalid structured output or validator rejection.
- Malformed or schema-invalid provider output is classified separately as
  invalid structured output so the module or pipeline can apply its own retry
  and rejection policy.
- Domain semantic checks, evidence decisions, and deterministic normalization
  run outside the provider adapter.

## Focused Verification

Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
tests when changing this boundary. Prompt changes also require the owning
module’s asset tests, and retry or debug changes require focused pipeline or
CLI coverage. The focused runtime and D&D checks are:

~~~sh
go test ./internal/framework/llm/... ./internal/modules/dnd/...
~~~