Refresh typed module and D&D contributor recipes
This commit is contained in:
@@ -1,195 +1,165 @@
|
||||
# Module Internals
|
||||
|
||||
This guide owns the mechanics for implementing and registering production
|
||||
modules. [Configuration](../config.md) owns selectable keys, binding syntax,
|
||||
reference configuration, and default validator chains. Durable input and output
|
||||
shapes belong in [integration contracts](../integrations/).
|
||||
This guide owns typed module construction, registration, and extension recipes.
|
||||
[Configuration](../config.md#production-module-keys) owns selectable keys,
|
||||
binding syntax, references, and default chains.
|
||||
[Integration contracts](../integrations/json-output.md) own durable shapes;
|
||||
[D&D Module Internals](dnd.md) owns that family's domain conventions.
|
||||
|
||||
The D&D family has additional shared conventions and domain-specific
|
||||
exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
||||
## Typed Boundary And Registration
|
||||
|
||||
## Module Boundary
|
||||
A module implements one fixed pipeline operation; an artifact family owns the
|
||||
related domain types, codec, modules, validators, prompts, and reference
|
||||
projections. Family ownership does not combine extraction/merge/normalization
|
||||
or turn the pipeline into a workflow language.
|
||||
|
||||
A module is a typed implementation registered for one pipeline stage. Its
|
||||
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||
stage, execution class, required and provided capabilities, artifact kind, and
|
||||
accepted reference slots. The execution class states whether a module is
|
||||
`deterministic` or `llm_backed`; registries retain it for catalog inspection and
|
||||
resolved-pipeline debug data without constructing the module. The framework
|
||||
uses the declaration to resolve a configured binding before it builds the
|
||||
implementation. After selection, the resolver applies profile inheritance only
|
||||
to bindings whose declared execution class is `llm_backed` and rejects a
|
||||
binding-specific profile on a deterministic module. The user-facing precedence
|
||||
contract belongs in [Configuration](../config.md#pipelines).
|
||||
[ModuleSpec](../../internal/framework/pipeline/module.go) declares the stable
|
||||
key, operation, execution class, artifact kind, capabilities, reference slots,
|
||||
and statically selected prompt IDs. Specs can be inspected without constructing
|
||||
modules. The resolver applies model-profile selections only to model roles.
|
||||
Static initial/correction dependencies use the same constants as request builders;
|
||||
they are preflight declarations, not asset manifests or execution fingerprints.
|
||||
|
||||
An eligible LLM-backed chunk, extract, merge, or normalize producer may also
|
||||
declare correction protocol `single_response_v1`. That declaration is a
|
||||
promise that the implementation accepts one attempt-local semantic correction
|
||||
and returns an owned copy of the exact one model response that directly
|
||||
controlled the candidate. It must forward correction only to its structured
|
||||
completion request; it must not manufacture prior-response material by
|
||||
serializing a normalized artifact or expose opaque application IDs. Input,
|
||||
output, validator, and deterministic specs cannot declare the protocol.
|
||||
[BuildRequest](../../internal/framework/pipeline/construction.go) supplies the
|
||||
shared native engine, option values, and prepared references. Option-aware
|
||||
modules provide a resolver validator and a builder using the same strict decoder.
|
||||
They do not rediscover configuration or reopen reference files. Generated
|
||||
references arrive through operation requests, not construction-time file reads.
|
||||
|
||||
Implementations that accept options must provide both an option validator and
|
||||
a builder. The validator is used while resolving configuration; the builder
|
||||
decodes the same options and constructs the implementation from the prepared
|
||||
`BuildRequest`. Reject unknown options in both paths. A builder receives only
|
||||
the dependencies and materialized references that the framework prepared for
|
||||
that operation, so it must not re-read configuration or files.
|
||||
Typed helpers retain the exact artifact type at registration:
|
||||
|
||||
Registry helpers register the typed builder for a stage-specific registry.
|
||||
They are preferable to hand-written untyped registration because they retain
|
||||
the artifact type at the framework boundary. Registrars validate the registries
|
||||
they need, register each leaf implementation, and add any family-owned assets
|
||||
or default validator chains. They return contextual errors so production
|
||||
composition fails at startup rather than at the first run.
|
||||
- [RegisterExtractorBuilder](../../internal/framework/pipeline/extractor_registry.go),
|
||||
[RegisterMergerBuilder](../../internal/framework/pipeline/merger_registry.go), and
|
||||
[RegisterNormalizerBuilder](../../internal/framework/pipeline/normalizer_registry.go)
|
||||
connect typed production to private framework erasure.
|
||||
- [Typed validators](../../internal/framework/pipeline/validator_registry.go)
|
||||
register compatible kind/type-specific assessment.
|
||||
- [Artifact codecs](../../internal/framework/pipeline/artifact_codec_registry.go)
|
||||
own durable encoding/decoding and intrinsic schema checks.
|
||||
- Optional [evidence projectors](../../internal/framework/pipeline/artifact_evidence_registry.go)
|
||||
copy direct generic source references using the codec's exact Go type.
|
||||
|
||||
A validator that returns a completed rejection must supply two separate
|
||||
bounded values: a stable `ReasonCode` for provenance and actionable
|
||||
`CorrectionGuidance` for the producer. Guidance identifies the semantic defect
|
||||
and the constraints on one complete corrected replacement. It must not contain
|
||||
validator keys, diagnostic paths, opaque application IDs, or other internal
|
||||
identifiers. An operator-facing `Message` may explain the same event, but the
|
||||
framework never copies it into a model request. Missing or invalid guidance is
|
||||
a validator contract failure.
|
||||
Preparation rejects incompatible types/registrations before parsing source.
|
||||
Projectors do not inspect arbitrary domain JSON or publish files; the
|
||||
[evidence contract](../integrations/evidence-context.md) owns the excerpt.
|
||||
|
||||
An artifact family can register an optional typed evidence projector alongside
|
||||
its codec. The projector returns defensive copies of the artifact's direct
|
||||
generic source references and must use the codec's exact Go type. It does not
|
||||
interpret surrounding context or publish files; the pipeline validates the
|
||||
capability during preparation and the output boundary owns publication. See
|
||||
the [Published Evidence Context contract](../integrations/evidence-context.md)
|
||||
for the durable source-unit excerpt. Lane artifacts retain citation and lane
|
||||
provenance; the framework does not add either to that published excerpt.
|
||||
## Native Producer Recipe
|
||||
|
||||
An artifact family is broader than a module: it owns the cohesive domain
|
||||
feature across its artifact type, codec, stage modules, validators, prompt
|
||||
policy, schemas, identity helpers, and reference projections. An extractor and
|
||||
normalizer in one artifact family remain independently registered modules in
|
||||
their respective pipeline stages. This ownership vocabulary does not create a
|
||||
new registry or change the fixed pipeline.
|
||||
Follow the actual [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
|
||||
alongside [typed requests/results](../../internal/framework/contracts/typed_pipeline.go)
|
||||
and [operation contracts](../../internal/framework/contracts/producer.go):
|
||||
|
||||
## Production Composition
|
||||
1. Validate operation input and prepare owned transcript/reference/contextual
|
||||
projections once. Retain those fixed bytes and resolution mappings locally.
|
||||
2. Build the initial `promptkit.RunRequest` using the declared prompt ID,
|
||||
operation profile/session, native artifacts, and variables.
|
||||
3. Call [RunProducer](../../internal/framework/pipeline/producer.go) with the shared
|
||||
engine and the supplied typed operation. Supply ordinary interpretation,
|
||||
correction-request, and supported fallback functions.
|
||||
4. Interpret `*promptkit.RunResult` into the typed domain candidate. Attach
|
||||
deterministic identities and canonical source references here, not by asking
|
||||
the model to reproduce opaque IDs. Return actionable interpretation guidance
|
||||
separately from operator diagnostics where the module can correct a selection.
|
||||
5. Build the follow-up request from new findings/assignment only. Preserve
|
||||
profile/session selections; native coordination carries prior conversation.
|
||||
Do not rerender source inputs, serialize a domain artifact as a previous
|
||||
assistant response, or implement another turn loop.
|
||||
6. Return the helper's value and outcome in the typed operation result. The
|
||||
runner has already supplied assessment; do not validate again after the helper.
|
||||
|
||||
Production composition is intentionally split by family:
|
||||
The helper checks native structural status before interpretation, binds native
|
||||
execution overrides/observers, assesses every delivered usable candidate, and
|
||||
handles prepared reuse. Modules do not replace its assessor/cache or infer
|
||||
acceptance from provider success. [Pipeline Internals](pipeline.md) owns the
|
||||
complete assessment/disposition algorithm; [LLM Integration](llm.md) owns native
|
||||
execution and [State Internals](state.md) owns storage/capture lifecycle.
|
||||
|
||||
- The generic registrar provides the unit chunker, generic JSON validators,
|
||||
JSON output encoder, and shared semantic-reconciliation prompt and response
|
||||
schema assets.
|
||||
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||
validators, prompt assets, fallback profile asset, and default chains. Its behavioral conventions
|
||||
are documented in [D&D Module Internals](dnd.md).
|
||||
Fallback construction is explicit and family-owned. It must satisfy intrinsic
|
||||
shape/source/downstream invariants, but does not rerun ordinary validators or
|
||||
claim semantic approval. Unsupported/invalid construction fails. No earlier
|
||||
candidate is recovered after an unusable continuation; cancellation produces no
|
||||
fallback. Modules do not pick a permissive default or hide an ordinary operation
|
||||
error as recovery.
|
||||
|
||||
The CLI owns the composition that invokes these registrars. A module package
|
||||
may register its own family but must not assemble the CLI or make framework
|
||||
packages depend on production extensions.
|
||||
## Validators And Deterministic Work
|
||||
|
||||
Validators receive native execution options and detached candidate/context views,
|
||||
not a producer operation or semantic-turn loop. Return approval, or rejection
|
||||
with separate stable reason and actionable `CorrectionGuidance`. Operator
|
||||
`Message` is not model feedback. The detecting rule owns guidance about fields
|
||||
the producer controls, without internal paths/keys/opaque identities.
|
||||
Invalid rejection guidance is a contract failure.
|
||||
|
||||
A model validator calls the shared native engine directly, checks native errors/
|
||||
validation before interpretation, and distinguishes unavailable judgment from
|
||||
semantic rejection while preserving causes. See the
|
||||
[combat validator](../../internal/modules/dnd/validate/scenedescriptions/combat_semantics/validator.go).
|
||||
Do not rerun a validator to extend native repair budgets.
|
||||
|
||||
Deterministic modules produce once through ordinary typed interfaces. The runner
|
||||
assesses the value and can invoke an explicitly supported result fallback after
|
||||
nonacceptance. Production errors propagate without generic retries. Model-backed
|
||||
modules with legitimate no-model branches use direct `AssessProducer` on that
|
||||
value; preparation-limit terminal handling uses `FinishProducerFailure` rather
|
||||
than synthetic requests. The [noop normalizer](../../internal/modules/generic/normalize/noop/normalizer.go)
|
||||
and registry normalizers show the distinct paths.
|
||||
|
||||
## Prompt And Schema Ownership
|
||||
|
||||
[assets/notarius.yml](../../assets/notarius.yml) imports ordinary native prompt
|
||||
declarations and profiles with their filesystem contexts. The content-only
|
||||
assets package has no registration behavior. Declarations own ordered messages,
|
||||
cache hints, default profiles, and output contracts/schema selection. Use real
|
||||
relative content-file paths; no virtual mounts or Go prompt manifests.
|
||||
|
||||
A producer's initial/correction pair requests complete responses with matching
|
||||
schema/profile defaults. Put enduring rules in initial system context and only
|
||||
current guidance/new assignment in follow-ups. Native structural repair remains
|
||||
upstream-owned. Private response envelopes are not durable artifact contracts.
|
||||
See [schema selection](../config.md#prompt-output-contracts-and-schemas) and the
|
||||
[D&D prompt recipe](dnd.md#prompt-construction) for concrete examples.
|
||||
|
||||
## Semantic Reconciliation
|
||||
|
||||
`internal/framework/semanticreconcile` is a domain-neutral strategy used by a
|
||||
typed normalize module; it is not itself a selectable stage module. A
|
||||
source-backed artifact-family normalizer projects its deterministic records
|
||||
into contextual candidates and owned typed record envelopes, supplies its
|
||||
chosen prompt identity and resolved LLM profile, and constructs an engine with
|
||||
explicit limits. The core filters invalid evidence, assigns contiguous
|
||||
one-based request-local candidate numbers, renders bounded candidate and
|
||||
transcript materials, invokes the structured-completion boundary, and assesses
|
||||
the returned duplicate groups into a stable non-overlapping plan. Candidate
|
||||
material declares the exact inclusive number range for the request. Transcript
|
||||
unit IDs and evidence range endpoints remain source positions in a separate
|
||||
namespace and are never valid candidate numbers.
|
||||
The [shared core](../../internal/framework/semanticreconcile/preparation.go)
|
||||
offers preparation, native request builders, proposal assessment, and typed
|
||||
application functions, not a replacement execution service.
|
||||
[Requests](../../internal/framework/semanticreconcile/requests.go) consume one
|
||||
retained preparation. [Proposal safety](../../internal/framework/semanticreconcile/proposal.go)
|
||||
resolves request-local handles; [application](../../internal/framework/semanticreconcile/application.go)
|
||||
clones/preserves originals and applies domain policies.
|
||||
|
||||
The normalizer then applies that plan through a typed `ApplicationPolicy`. The
|
||||
core preserves ungrouped records, contribution order, and provenance while the
|
||||
artifact family owns group guards, field and evidence consolidation, durable
|
||||
ID derivation, retry and fallback presentation, classified diagnostics, and
|
||||
postconditions.
|
||||
These candidate numbers are the concrete private representation of ADR-0013's
|
||||
request-local handles; they do not enter the typed value or durable artifact.
|
||||
Fewer than two eligible candidates skips model invocation; exceeding a
|
||||
candidate or combined-material bound preserves the deterministic result under
|
||||
the family's fallback policy. Provider, transport, cancellation, and
|
||||
context-construction failures remain execution errors.
|
||||
The [item normalizer](../../internal/modules/dnd/normalize/itemregistry/normalizer.go)
|
||||
preprocesses fixed originals once, prepares bounded windows/handles once, then
|
||||
uses the same producer helper as extraction. Each complete proposal is assessed
|
||||
and applied afresh to originals before pipeline assessment. Domain guards,
|
||||
identity, evidence, diagnostics, and supported final-only fallback remain with
|
||||
the artifact family. Do not union plans across turns, reuse prior mutated output,
|
||||
or introduce module retry/repair protocols. [D&D reconciliation](dnd.md#semantic-registry-reconciliation)
|
||||
explains the concrete policy differences.
|
||||
|
||||
When the engine actually makes a proposal call, its typed result carries the
|
||||
owned exact proposal response under the same correction contract as other
|
||||
eligible producers. Deterministic skip, limit, and fallback outcomes carry no
|
||||
model candidate, so a later rejection applies terminal policy without spending
|
||||
an ineffective semantic retry.
|
||||
## Production Composition And Change Checklist
|
||||
|
||||
When proposal assessment or typed group application rejects a structurally
|
||||
valid group, the normalizer may return its safe partial value with a
|
||||
module-requested retry. A feedback-capable directive supplies bounded
|
||||
model-facing correction guidance separately from operator diagnostics and
|
||||
retains the exact proposal response as its candidate. The shared stage retry
|
||||
mechanism appends that response and guidance to a fresh complete request. A
|
||||
structurally invalid completion has no valid candidate and therefore requests a
|
||||
feedback-free fresh attempt. On exhaustion, only the final safe fallback and
|
||||
its bounded process diagnostic advance to validation.
|
||||
The CLI invokes generic, Seriatim, and D&D registrars. They register typed modules,
|
||||
codecs, evidence, and owned validator defaults, not prompts/schemas/profiles.
|
||||
[Seriatim](../integrations/seriatim.md) retains input-format ownership; generic
|
||||
operations remain domain-neutral; D&D does not compose process behavior.
|
||||
|
||||
The core supplies a conservative generic prompt and the single private
|
||||
response schema. A domain prompt may substitute its semantic instructions but
|
||||
mounts the core-owned protocol and candidate/transcript presentation assets.
|
||||
Prompt, schema, policy, and limit identities participate in manifest metadata
|
||||
and checkpoint fingerprints. The generic registrar owns production
|
||||
registration of those shared assets; a consuming domain registrar owns only
|
||||
its domain prompt.
|
||||
For an extension, add the typed domain/codec boundary, strict options and spec,
|
||||
native declarations/static dependencies where needed, stage implementation,
|
||||
and owning registrar entry. Add default chains only where the family owns them.
|
||||
Reference slots declare artifact/media/size requirements; generated handoffs
|
||||
must retain prior-step compatibility. Preserve ownership of request maps/slices
|
||||
and returned stored data. Modules receive no physical roots or CLI reporters.
|
||||
|
||||
## Adding Or Changing A Module
|
||||
|
||||
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||
input or durable artifact formats in the relevant integration contract,
|
||||
not in this guide or in a private LLM response type.
|
||||
2. Define a stable `ModuleSpec` with an explicit execution class, the exact
|
||||
capabilities, and reference slots needed for the operation. Model a
|
||||
producer/consumer handoff as an artifact-compatible slot; configuration
|
||||
then chooses an external file or a generated binding.
|
||||
3. Implement strict option decoding, construction, and the typed stage
|
||||
interface. Preserve caller ownership: do not retain mutable request data
|
||||
and return defensive copies where an implementation exposes stored data.
|
||||
If declaring correction capability, forward the request correction and
|
||||
retain only the exact validated response that controlled the result.
|
||||
4. Register the module through its typed registry helper and add it to the
|
||||
owning family registrar. Add a default validator chain only when that
|
||||
family owns the behavior; otherwise require an explicit compatible chain.
|
||||
Every rejection path in a validator must provide actionable correction
|
||||
guidance while retaining its stable internal reason code.
|
||||
5. Update the selectable-key and chain reference in
|
||||
[Configuration](../config.md#production-module-keys), the applicable
|
||||
integration contract, and focused tests. Keep the configuration document
|
||||
as the sole list of production keys and validator order.
|
||||
|
||||
## Validation And References
|
||||
|
||||
Validators operate on the value produced at their configured stage. A default
|
||||
chain is ordered behavior, not a set: JSON parsing, structural checks,
|
||||
domain-specific checks, durable-schema checks, and advisory checks may have
|
||||
different responsibilities and failure handling. The active default chains and
|
||||
override rules are maintained in
|
||||
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||
|
||||
Reference slots are part of the module specification. They describe the
|
||||
accepted artifact kind, media type, size, and whether a binding is required;
|
||||
the framework validates those constraints before construction. An external
|
||||
reference is materialized during preparation. A generated reference is a
|
||||
compatible normalized artifact handed from an earlier pipeline step at
|
||||
operation time. The configuration reference rules, including precedence and
|
||||
ordered-handoff requirements, are maintained in
|
||||
[Configuration](../config.md#references-and-ordered-handoffs).
|
||||
|
||||
## Focused Verification
|
||||
|
||||
Exercise the leaf implementation and its registration path when changing a
|
||||
module. Registry and registrar tests cover duplicate keys, required registries,
|
||||
and typed construction; pipeline resolution tests cover capabilities, options,
|
||||
and reference compatibility. Domain packages should additionally test their
|
||||
codecs, validators, normalizers, and any integration handoffs they own.
|
||||
|
||||
Run the affected package tests while iterating. The complete module suite is:
|
||||
|
||||
~~~sh
|
||||
go test ./internal/modules/...
|
||||
~~~
|
||||
Update canonical [keys/chains](../config.md#production-validator-keys-and-default-chains)
|
||||
and durable contracts when their behavior changes; complete runnable configurations
|
||||
belong in [examples](../../examples/). Test the actual leaf and registration path,
|
||||
exact-type compatibility, domain invariants, fixed-input correction, unavailable/
|
||||
failed outcomes, supported fallback, and cancellation where applicable.
|
||||
[Spell tests](../../internal/modules/dnd/extract/spells/extractor_test.go),
|
||||
[item normalization tests](../../internal/modules/dnd/normalize/itemregistry/normalizer_test.go),
|
||||
and [registration dependencies](../../internal/modules/dnd/register/dependencies_test.go)
|
||||
are concrete starting points. Use real kits with native provider fakes, not a
|
||||
replacement completion mock or full prompt-prose snapshots.
|
||||
[Development](../development.md#validation) owns wider checks.
|
||||
|
||||
Reference in New Issue
Block a user