Refresh typed module and D&D contributor recipes

This commit is contained in:
2026-11-22 06:48:55 +00:00
parent 41f6079040
commit aa4ce04683
3 changed files with 369 additions and 467 deletions

View File

@@ -1,195 +1,165 @@
# Module Internals
This guide owns the mechanics for implementing and registering production
modules. [Configuration](../config.md) owns selectable keys, binding syntax,
reference configuration, and default validator chains. Durable input and output
shapes belong in [integration contracts](../integrations/).
This guide owns typed module construction, registration, and extension recipes.
[Configuration](../config.md#production-module-keys) owns selectable keys,
binding syntax, references, and default chains.
[Integration contracts](../integrations/json-output.md) own durable shapes;
[D&D Module Internals](dnd.md) owns that family's domain conventions.
The D&D family has additional shared conventions and domain-specific
exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
## Typed Boundary And Registration
## Module Boundary
A module implements one fixed pipeline operation; an artifact family owns the
related domain types, codec, modules, validators, prompts, and reference
projections. Family ownership does not combine extraction/merge/normalization
or turn the pipeline into a workflow language.
A module is a typed implementation registered for one pipeline stage. Its
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
stage, execution class, required and provided capabilities, artifact kind, and
accepted reference slots. The execution class states whether a module is
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
resolved-pipeline debug data without constructing the module. The framework
uses the declaration to resolve a configured binding before it builds the
implementation. After selection, the resolver applies profile inheritance only
to bindings whose declared execution class is `llm_backed` and rejects a
binding-specific profile on a deterministic module. The user-facing precedence
contract belongs in [Configuration](../config.md#pipelines).
[ModuleSpec](../../internal/framework/pipeline/module.go) declares the stable
key, operation, execution class, artifact kind, capabilities, reference slots,
and statically selected prompt IDs. Specs can be inspected without constructing
modules. The resolver applies model-profile selections only to model roles.
Static initial/correction dependencies use the same constants as request builders;
they are preflight declarations, not asset manifests or execution fingerprints.
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
declare correction protocol `single_response_v1`. That declaration is a
promise that the implementation accepts one attempt-local semantic correction
and returns an owned copy of the exact one model response that directly
controlled the candidate. It must forward correction only to its structured
completion request; it must not manufacture prior-response material by
serializing a normalized artifact or expose opaque application IDs. Input,
output, validator, and deterministic specs cannot declare the protocol.
[BuildRequest](../../internal/framework/pipeline/construction.go) supplies the
shared native engine, option values, and prepared references. Option-aware
modules provide a resolver validator and a builder using the same strict decoder.
They do not rediscover configuration or reopen reference files. Generated
references arrive through operation requests, not construction-time file reads.
Implementations that accept options must provide both an option validator and
a builder. The validator is used while resolving configuration; the builder
decodes the same options and constructs the implementation from the prepared
`BuildRequest`. Reject unknown options in both paths. A builder receives only
the dependencies and materialized references that the framework prepared for
that operation, so it must not re-read configuration or files.
Typed helpers retain the exact artifact type at registration:
Registry helpers register the typed builder for a stage-specific registry.
They are preferable to hand-written untyped registration because they retain
the artifact type at the framework boundary. Registrars validate the registries
they need, register each leaf implementation, and add any family-owned assets
or default validator chains. They return contextual errors so production
composition fails at startup rather than at the first run.
- [RegisterExtractorBuilder](../../internal/framework/pipeline/extractor_registry.go),
[RegisterMergerBuilder](../../internal/framework/pipeline/merger_registry.go), and
[RegisterNormalizerBuilder](../../internal/framework/pipeline/normalizer_registry.go)
connect typed production to private framework erasure.
- [Typed validators](../../internal/framework/pipeline/validator_registry.go)
register compatible kind/type-specific assessment.
- [Artifact codecs](../../internal/framework/pipeline/artifact_codec_registry.go)
own durable encoding/decoding and intrinsic schema checks.
- Optional [evidence projectors](../../internal/framework/pipeline/artifact_evidence_registry.go)
copy direct generic source references using the codec's exact Go type.
A validator that returns a completed rejection must supply two separate
bounded values: a stable `ReasonCode` for provenance and actionable
`CorrectionGuidance` for the producer. Guidance identifies the semantic defect
and the constraints on one complete corrected replacement. It must not contain
validator keys, diagnostic paths, opaque application IDs, or other internal
identifiers. An operator-facing `Message` may explain the same event, but the
framework never copies it into a model request. Missing or invalid guidance is
a validator contract failure.
Preparation rejects incompatible types/registrations before parsing source.
Projectors do not inspect arbitrary domain JSON or publish files; the
[evidence contract](../integrations/evidence-context.md) owns the excerpt.
An artifact family can register an optional typed evidence projector alongside
its codec. The projector returns defensive copies of the artifact's direct
generic source references and must use the codec's exact Go type. It does not
interpret surrounding context or publish files; the pipeline validates the
capability during preparation and the output boundary owns publication. See
the [Published Evidence Context contract](../integrations/evidence-context.md)
for the durable source-unit excerpt. Lane artifacts retain citation and lane
provenance; the framework does not add either to that published excerpt.
## Native Producer Recipe
An artifact family is broader than a module: it owns the cohesive domain
feature across its artifact type, codec, stage modules, validators, prompt
policy, schemas, identity helpers, and reference projections. An extractor and
normalizer in one artifact family remain independently registered modules in
their respective pipeline stages. This ownership vocabulary does not create a
new registry or change the fixed pipeline.
Follow the actual [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
alongside [typed requests/results](../../internal/framework/contracts/typed_pipeline.go)
and [operation contracts](../../internal/framework/contracts/producer.go):
## Production Composition
1. Validate operation input and prepare owned transcript/reference/contextual
projections once. Retain those fixed bytes and resolution mappings locally.
2. Build the initial `promptkit.RunRequest` using the declared prompt ID,
operation profile/session, native artifacts, and variables.
3. Call [RunProducer](../../internal/framework/pipeline/producer.go) with the shared
engine and the supplied typed operation. Supply ordinary interpretation,
correction-request, and supported fallback functions.
4. Interpret `*promptkit.RunResult` into the typed domain candidate. Attach
deterministic identities and canonical source references here, not by asking
the model to reproduce opaque IDs. Return actionable interpretation guidance
separately from operator diagnostics where the module can correct a selection.
5. Build the follow-up request from new findings/assignment only. Preserve
profile/session selections; native coordination carries prior conversation.
Do not rerender source inputs, serialize a domain artifact as a previous
assistant response, or implement another turn loop.
6. Return the helper's value and outcome in the typed operation result. The
runner has already supplied assessment; do not validate again after the helper.
Production composition is intentionally split by family:
The helper checks native structural status before interpretation, binds native
execution overrides/observers, assesses every delivered usable candidate, and
handles prepared reuse. Modules do not replace its assessor/cache or infer
acceptance from provider success. [Pipeline Internals](pipeline.md) owns the
complete assessment/disposition algorithm; [LLM Integration](llm.md) owns native
execution and [State Internals](state.md) owns storage/capture lifecycle.
- The generic registrar provides the unit chunker, generic JSON validators,
JSON output encoder, and shared semantic-reconciliation prompt and response
schema assets.
- The Seriatim registrar provides the transcript input adapter. Its external
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
are documented in [D&D Module Internals](dnd.md).
Fallback construction is explicit and family-owned. It must satisfy intrinsic
shape/source/downstream invariants, but does not rerun ordinary validators or
claim semantic approval. Unsupported/invalid construction fails. No earlier
candidate is recovered after an unusable continuation; cancellation produces no
fallback. Modules do not pick a permissive default or hide an ordinary operation
error as recovery.
The CLI owns the composition that invokes these registrars. A module package
may register its own family but must not assemble the CLI or make framework
packages depend on production extensions.
## Validators And Deterministic Work
Validators receive native execution options and detached candidate/context views,
not a producer operation or semantic-turn loop. Return approval, or rejection
with separate stable reason and actionable `CorrectionGuidance`. Operator
`Message` is not model feedback. The detecting rule owns guidance about fields
the producer controls, without internal paths/keys/opaque identities.
Invalid rejection guidance is a contract failure.
A model validator calls the shared native engine directly, checks native errors/
validation before interpretation, and distinguishes unavailable judgment from
semantic rejection while preserving causes. See the
[combat validator](../../internal/modules/dnd/validate/scenedescriptions/combat_semantics/validator.go).
Do not rerun a validator to extend native repair budgets.
Deterministic modules produce once through ordinary typed interfaces. The runner
assesses the value and can invoke an explicitly supported result fallback after
nonacceptance. Production errors propagate without generic retries. Model-backed
modules with legitimate no-model branches use direct `AssessProducer` on that
value; preparation-limit terminal handling uses `FinishProducerFailure` rather
than synthetic requests. The [noop normalizer](../../internal/modules/generic/normalize/noop/normalizer.go)
and registry normalizers show the distinct paths.
## Prompt And Schema Ownership
[assets/notarius.yml](../../assets/notarius.yml) imports ordinary native prompt
declarations and profiles with their filesystem contexts. The content-only
assets package has no registration behavior. Declarations own ordered messages,
cache hints, default profiles, and output contracts/schema selection. Use real
relative content-file paths; no virtual mounts or Go prompt manifests.
A producer's initial/correction pair requests complete responses with matching
schema/profile defaults. Put enduring rules in initial system context and only
current guidance/new assignment in follow-ups. Native structural repair remains
upstream-owned. Private response envelopes are not durable artifact contracts.
See [schema selection](../config.md#prompt-output-contracts-and-schemas) and the
[D&D prompt recipe](dnd.md#prompt-construction) for concrete examples.
## Semantic Reconciliation
`internal/framework/semanticreconcile` is a domain-neutral strategy used by a
typed normalize module; it is not itself a selectable stage module. A
source-backed artifact-family normalizer projects its deterministic records
into contextual candidates and owned typed record envelopes, supplies its
chosen prompt identity and resolved LLM profile, and constructs an engine with
explicit limits. The core filters invalid evidence, assigns contiguous
one-based request-local candidate numbers, renders bounded candidate and
transcript materials, invokes the structured-completion boundary, and assesses
the returned duplicate groups into a stable non-overlapping plan. Candidate
material declares the exact inclusive number range for the request. Transcript
unit IDs and evidence range endpoints remain source positions in a separate
namespace and are never valid candidate numbers.
The [shared core](../../internal/framework/semanticreconcile/preparation.go)
offers preparation, native request builders, proposal assessment, and typed
application functions, not a replacement execution service.
[Requests](../../internal/framework/semanticreconcile/requests.go) consume one
retained preparation. [Proposal safety](../../internal/framework/semanticreconcile/proposal.go)
resolves request-local handles; [application](../../internal/framework/semanticreconcile/application.go)
clones/preserves originals and applies domain policies.
The normalizer then applies that plan through a typed `ApplicationPolicy`. The
core preserves ungrouped records, contribution order, and provenance while the
artifact family owns group guards, field and evidence consolidation, durable
ID derivation, retry and fallback presentation, classified diagnostics, and
postconditions.
These candidate numbers are the concrete private representation of ADR-0013's
request-local handles; they do not enter the typed value or durable artifact.
Fewer than two eligible candidates skips model invocation; exceeding a
candidate or combined-material bound preserves the deterministic result under
the family's fallback policy. Provider, transport, cancellation, and
context-construction failures remain execution errors.
The [item normalizer](../../internal/modules/dnd/normalize/itemregistry/normalizer.go)
preprocesses fixed originals once, prepares bounded windows/handles once, then
uses the same producer helper as extraction. Each complete proposal is assessed
and applied afresh to originals before pipeline assessment. Domain guards,
identity, evidence, diagnostics, and supported final-only fallback remain with
the artifact family. Do not union plans across turns, reuse prior mutated output,
or introduce module retry/repair protocols. [D&D reconciliation](dnd.md#semantic-registry-reconciliation)
explains the concrete policy differences.
When the engine actually makes a proposal call, its typed result carries the
owned exact proposal response under the same correction contract as other
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
model candidate, so a later rejection applies terminal policy without spending
an ineffective semantic retry.
## Production Composition And Change Checklist
When proposal assessment or typed group application rejects a structurally
valid group, the normalizer may return its safe partial value with a
module-requested retry. A feedback-capable directive supplies bounded
model-facing correction guidance separately from operator diagnostics and
retains the exact proposal response as its candidate. The shared stage retry
mechanism appends that response and guidance to a fresh complete request. A
structurally invalid completion has no valid candidate and therefore requests a
feedback-free fresh attempt. On exhaustion, only the final safe fallback and
its bounded process diagnostic advance to validation.
The CLI invokes generic, Seriatim, and D&D registrars. They register typed modules,
codecs, evidence, and owned validator defaults, not prompts/schemas/profiles.
[Seriatim](../integrations/seriatim.md) retains input-format ownership; generic
operations remain domain-neutral; D&D does not compose process behavior.
The core supplies a conservative generic prompt and the single private
response schema. A domain prompt may substitute its semantic instructions but
mounts the core-owned protocol and candidate/transcript presentation assets.
Prompt, schema, policy, and limit identities participate in manifest metadata
and checkpoint fingerprints. The generic registrar owns production
registration of those shared assets; a consuming domain registrar owns only
its domain prompt.
For an extension, add the typed domain/codec boundary, strict options and spec,
native declarations/static dependencies where needed, stage implementation,
and owning registrar entry. Add default chains only where the family owns them.
Reference slots declare artifact/media/size requirements; generated handoffs
must retain prior-step compatibility. Preserve ownership of request maps/slices
and returned stored data. Modules receive no physical roots or CLI reporters.
## Adding Or Changing A Module
1. Choose the pipeline stage and the typed artifact boundary. Put external
input or durable artifact formats in the relevant integration contract,
not in this guide or in a private LLM response type.
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
capabilities, and reference slots needed for the operation. Model a
producer/consumer handoff as an artifact-compatible slot; configuration
then chooses an external file or a generated binding.
3. Implement strict option decoding, construction, and the typed stage
interface. Preserve caller ownership: do not retain mutable request data
and return defensive copies where an implementation exposes stored data.
If declaring correction capability, forward the request correction and
retain only the exact validated response that controlled the result.
4. Register the module through its typed registry helper and add it to the
owning family registrar. Add a default validator chain only when that
family owns the behavior; otherwise require an explicit compatible chain.
Every rejection path in a validator must provide actionable correction
guidance while retaining its stable internal reason code.
5. Update the selectable-key and chain reference in
[Configuration](../config.md#production-module-keys), the applicable
integration contract, and focused tests. Keep the configuration document
as the sole list of production keys and validator order.
## Validation And References
Validators operate on the value produced at their configured stage. A default
chain is ordered behavior, not a set: JSON parsing, structural checks,
domain-specific checks, durable-schema checks, and advisory checks may have
different responsibilities and failure handling. The active default chains and
override rules are maintained in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Reference slots are part of the module specification. They describe the
accepted artifact kind, media type, size, and whether a binding is required;
the framework validates those constraints before construction. An external
reference is materialized during preparation. A generated reference is a
compatible normalized artifact handed from an earlier pipeline step at
operation time. The configuration reference rules, including precedence and
ordered-handoff requirements, are maintained in
[Configuration](../config.md#references-and-ordered-handoffs).
## Focused Verification
Exercise the leaf implementation and its registration path when changing a
module. Registry and registrar tests cover duplicate keys, required registries,
and typed construction; pipeline resolution tests cover capabilities, options,
and reference compatibility. Domain packages should additionally test their
codecs, validators, normalizers, and any integration handoffs they own.
Run the affected package tests while iterating. The complete module suite is:
~~~sh
go test ./internal/modules/...
~~~
Update canonical [keys/chains](../config.md#production-validator-keys-and-default-chains)
and durable contracts when their behavior changes; complete runnable configurations
belong in [examples](../../examples/). Test the actual leaf and registration path,
exact-type compatibility, domain invariants, fixed-input correction, unavailable/
failed outcomes, supported fallback, and cancellation where applicable.
[Spell tests](../../internal/modules/dnd/extract/spells/extractor_test.go),
[item normalization tests](../../internal/modules/dnd/normalize/itemregistry/normalizer_test.go),
and [registration dependencies](../../internal/modules/dnd/register/dependencies_test.go)
are concrete starting points. Use real kits with native provider fakes, not a
replacement completion mock or full prompt-prose snapshots.
[Development](../development.md#validation) owns wider checks.