Files
notarius/docs/policy/architecture.md

390 lines
21 KiB
Markdown

# Architecture Policy
**Base Policy Version:** 1.3
**Last Updated**: 2026-11-22
**Upstream Source**: https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/architecture.md
The source above identifies the adopted baseline. Local native-kit architecture
decisions below are repository adaptations, not a new upstream policy adoption.
## Purpose And Ownership
The adopted policy defines the application's purpose and the architectural
constraints contributors must follow. It owns system boundaries, responsibility
and dependency rules, established invariants, and explicit non-goals.
Describe concrete package inventory and implementation mechanics in internal
documentation. Link to canonical public contracts for exact behavior and to
ADRs for decision rationale.
## Architectural Approach
Hexagonal architecture is strongly recommended. Keep pure domain logic independent
of external integration details and make dependency direction clear. Native kit
types are an intentional contract for model execution, configuration composition,
and CLI presentation; do not hide them behind a replacement completion API.
Implement separation with proportionate, idiomatic code. Choose interfaces and
package layout to serve demonstrated boundaries. Use adapters where they serve
an integration need, not merely to rename an adopted library's public types.
## Adopted Constraints And Implemented Structure
Architectural constraints govern current and future development. Record
implemented structure accurately, proposals and implementation status in the
roadmap, and significant decisions and rationale in ADRs. Describe a target
structure as implemented when its changes land.
Constraints must reflect deliberately selected requirements and decisions.
Do not promote incidental implementation choices into architectural invariants.
Notarius's fixed pipeline, artifact-family boundaries, and deployment-owned
cache permissions remain established product decisions. Successor ADRs
[0021](../adr/0021-compose-native-kits-and-prompt-declared-resources.md),
[0022](../adr/0022-use-native-continuation-and-application-assessment.md),
[0023](../adr/0023-unify-accepted-artifact-and-plan-caching.md), and
[0024](../adr/0024-publish-artifacts-with-native-reports-and-debug-evidence.md)
record the native cutover. Earlier accepted bodies remain historical decisions,
with status links identifying superseded portions.
## System Shape
Notarius is a Go application for configured structured extraction workflows.
Source formats, artifact domains, validation, and output encoding remain explicit
boundaries.
The application has one fixed pipeline shape:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Pipelines are configured compositions of this shape, not arbitrary DAGs or a
general workflow language. Every stage remains explicit; chunking, merging, or
normalization must not be hidden inside an extractor.
A stage module implements one pipeline stage. An artifact family owns a cohesive
domain feature across its stages, codecs, validators, prompts, identity rules,
and reference projections. Family ownership does not combine stages.
Input and chunking are pipeline-wide. Each selected artifact lane owns its
extract, merge, and normalize stages. Output aggregates the run's domain results.
Ordered steps support generated references without changing the fixed shape.
Notarius is contract-first without being abstraction-heavy. Extension points
protect demonstrated boundaries; new abstraction is not itself an architectural
goal. [Agent Policy](agents.md) owns implementation discipline.
## Layers And Dependency Direction
The composition root may depend on concrete implementations. Domain-neutral
models and framework layers provide reusable contracts and orchestration.
Concrete input, pipeline, output, and validation extensions depend inward on
those generic layers. Generic layers must not depend on production extensions;
extensions must not compose the application or own process behavior.
Semantic reconciliation is a domain-neutral mechanism for fixed candidate
preparation, native request construction, proposal decoding, and safe typed
application. It does not own domain identity, durable IDs, warning semantics,
or artifact construction rules. Native execution uses the shared producer path.
The root `assets` package remains a content-only dependency leaf. It may expose
a read-only embedded filesystem but contains no business logic or dependencies
on internal packages or PromptKit. Native source assembly retains its filesystem
context; the package is not a behavioral registry or public extension contract.
This retains [ADR-0011](../adr/0011-centralize-llm-assets.md)'s content boundary.
The following dependency boundaries are mandatory:
- extractors and validators do not depend on concrete input adapters;
- provider-specific transport and authentication remain inside PromptKit;
- model-executing framework code, modules, and validators use native PromptKit
requests, results, exchanges, errors, and observations directly;
- pure domain types, codecs, reference resolution, and deterministic helpers do
not acquire PromptKit dependencies merely because surrounding producers use it;
- other external dependency types cross internal boundaries only where that
dependency is an explicit contract.
Shared helpers support demonstrated common needs without moving source-format
or domain knowledge into generic packages. Dependencies require a clear
correctness, security, interoperability, or complexity benefit.
[Internal Overview](../internal/overview.md) owns the concrete package inventory.
## Source And Domain Boundaries
Input modules translate external formats into generic source documents, units,
and references. Format-specific fields, schemas, and validation remain with the
input module and its integration contract.
A source reference identifies an ordered range of generic units. Framework code
preserves ranges unless a module explicitly owns their transformation. Every
source unit carries a validated self-reference to its document and unit ID.
Auxiliary references provide context or disambiguation, not source evidence;
they must not be converted into source references.
Modules own domain semantics, prompt selection, interpretation, deterministic
identity, and supported fallback construction. Physical asset centralization does
not transfer semantic ownership. Native prompt declarations own default output
contracts and schema selections; PromptKit prepares schemas and performs
structural validation and repair. Modules declare small static resource
dependencies for preflight, not manifests, schema registries, or fingerprints.
Intrinsic durable artifact schemas remain application contracts.
Typed artifact registrations declare one stable kind and exact Go type through
extraction, merge, normalization, and semantic validation. Resolution requires a
compatible codec and matching kind-specific variants. Private framework erasure
reports incompatibility as an error rather than a panic.
A kind may provide a typed evidence projection copying direct generic source
references. Preparation proves its exact codec-type compatibility. Evidence is
reconstructed from serialized normalized artifacts, including supported fallbacks,
without generic code inspecting domain JSON or importing domain types. The output
boundary owns publication.
When a model selects an entity, modules supply contextual selections and attach
opaque application IDs deterministically after exact resolution. Models do not
receive or reproduce opaque application IDs. Reconciliation may expose
request-local candidate handles, resolved before typed application and never
used as durable identity. Retain
[ADR-0012](../adr/0012-resolve-opaque-entity-identifiers-deterministically.md) and
[ADR-0013](../adr/0013-use-request-local-candidate-handles-for-semantic-reconciliation.md)'s
identity and safety decisions.
## Pipeline Composition And Ownership
Configuration and registries select modules, explicit defaults, references, and
runtime overrides. Validate selected module and validator options before
execution. Preparation constructs the complete implementation set; the runner
accepts only that prepared set. Construction failures precede source parsing.
Stage ownership remains explicit:
- input translates external material into the generic source model;
- chunk produces source-addressed plans;
- extract produces artifacts from chunks;
- merge combines extraction outputs;
- normalize reconciles merged output;
- output encodes domain results and enabled auxiliaries into logical files.
The framework materializes plans into chunks before candidate assessment and
lane execution. LLM plans use the accepted-result cache; hits rematerialize
against the current source without new semantic assessment. Source identity alone
does not select a canonical partition. Deterministic chunkers execute normally.
The framework owns orchestration, origin enrichment, aggregation, and handoff
provenance. Modules return logical results and classified diagnostics, not CLI
reports, physical roots, durable placement, or capture lifecycle.
After pipeline-wide chunking, one pool bounds ready extraction and ready
merge/normalize lane-completion jobs together. Extraction dispatch uses stable
chunk-first, lane-second order. A continuation becomes ready after its lane's
extraction finishes; a worker does not wait for prerequisites while holding a
slot. Validation and correction run inside the current job without another
worker acquisition. Ordered steps retain their barriers. Do not create unbounded
goroutines per lane or chunk.
Completion timing does not choose public ordering or terminal errors. The
coordinator orders artifacts, grouped diagnostics, and errors by stable scope.
Cancellation stops dispatch, drains started work, and releases workers and native
admission before returning.
Warnings are process signals: configuration degradation, qualified fallback,
or incomplete configured work. Quality and grounding findings are advisories;
accepted cleanup is observation. Modules own classification, while the framework
adds origin and aggregates bounded groups. A non-degraded successful run has no
actionable warnings. Retain
[ADR-0015](../adr/0015-separate-process-warnings-from-quality-diagnostics.md)'s
classification decision, not its retired durable reporting contracts.
## Validation And Disposition
Validation is a framework-managed boundary around chunk, extract, merge, and
normalize candidates. Validators receive immutable views and return approve,
reject, fail, or unavailable findings. Typed validators receive domain values;
chunk validators receive generic chunks; serialized validators receive bytes
and schema metadata. Incompatible targets or kinds cannot substitute for one
another.
Assess each usable initial or revised candidate once through every applicable
validator in configured order. Rejections do not short-circuit later validators.
Earlier approvals and unchanged revisions do not excuse reassessment; the final
allowed response also receives the whole chain. Cancellation or fatal framework
failures can interrupt assessment. There is no duplicate final assessment.
Actionable correction guidance is distinct from reason codes, operator messages,
and failures. The detecting validator or module owns it. The pipeline aggregates
it in order; a native module template supplies the correction assignment. Missing
required guidance is a contract failure, not an invitation to invent instructions.
Unavailable assessment is not a semantic defect or ordinary acceptance.
The producer's resolved mode owns terminal disposition. Strict requires
acceptance; explicitly selected permissive mode may construct a supported
fallback. A fallback must satisfy intrinsic shape and downstream invariants but
does not rerun the ordinary validator chain or claim semantic approval. Unsupported
or invalid fallback construction fails normally. Caller cancellation never
initiates fallback. Exact mode and turn controls belong in [Configuration](../config.md).
Default validator chains remain central production composition policy.
Configuration can replace a stage-local default, including with an explicit empty
chain. Configured order is authoritative. Intrinsic codec/source correctness
still applies with an empty chain.
## Native Execution Boundary
The composition root constructs one shared native PromptKit engine from the
application's ConfigKit result. All model roles use that engine. PromptKit owns
transport, authentication, transient retries, structural validation and repair,
conversation history, turn iteration, and backend admission. Notarius does not
pin omitted native budgets or add outer retries to extend them. Deterministic
operations have no generic retry budget or synthetic model request.
One small typed producer helper connects module-owned fixed inputs, native
request builders, interpretation, and supported fallback functions with the
pipeline assessor. Initial/correction pairs use complete replacement responses
and the same declared schema. Enduring principles belong in initial system
context; correction supplies current findings and a new assignment. Native
coordination carries the growing exchange without manual response replay or
regeneration of original references. Normalization applies each proposal afresh
to fixed originals, without cross-turn unions.
Check native errors and validation status before interpretation, including failed
validation with nil Go error. Native success, stopped, or exhausted status alone
does not establish domain acceptance. Do not recover an earlier candidate after
a result-less continuation failure or assess an undelivered response. LLM
validators execute once per assessment, without a separate semantic correction
loop. Calls respect cancellation and applicable native timeouts.
Worker limits and native backend admission are independent. Profiles selecting
one backend ID share its engine capacity; different IDs do not imply an endpoint
limit. Notarius injects no backend concurrency/queue defaults and has no
cross-backend scheduler, second admission queue, or capacity retry loop. Completed
turns release admission before assessment so nested validators can use the same
limited backend. Native failures retain their causes and follow the producer's
mode; they do not become correction guidance.
## Configuration And Provenance
ConfigKit owns loading, imports, composition, overlays, inheritance, source
contexts, provenance, and secret-aware inspection. Notarius owns application
meaning through supported definition rules and domain resolution, retaining
[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
`NOTARIUS_CONFIG` is discovery only; field-level environment overlays are not
application configuration. Named pipelines keep structural composition explicit.
One application-owned result supplies engine construction and whole-app/focused
inspection. Preserve source contexts and captured selection evidence rather than
reparsing settings or maintaining duplicate loaders. Request overrides retain
native supplied-value semantics. CommandKit, ckconfig, and ckprompt own standard
commands, parsing, presentation, and delivery outcomes. Notarius supplies domain
commands and report content, not a universal result envelope.
Offline preflight shares domain resolution and checks statically known module,
reference, initial/correction/validator prompt, and applicable profile dependencies
before execution. It requires no transcript, reference bytes, credentials,
provider call, or run storage. Dynamic selections remain runtime checks.
Shared loaders allocate no run storage; capture belongs to the run path.
## Trust Boundaries And Threat Scope
Source material and model responses cross application validation boundaries.
Host access policy belongs to deployments. Hypothetical privileged actors
concurrently modifying the filesystem do not justify new application defenses
without an explicit decision. This does not remove adopted correctness boundaries
or claim excluded scenarios cannot occur. Raise new threats or guarantees under
[Agent Policy](agents.md).
Configuration, operations, and internal documents own credential supply,
runtime handling, and mechanisms under [Documentation Policy](documentation.md).
## State, Output, And Safety
Output, cache, and debug retain independent roots and lifecycles:
- output is durable domain data; modules encode logical files and the CLI places them;
- cache is reconstructible accepted-artifact and LLM-plan state;
- debug is explicitly requested run/operation evidence, not a consumer API.
Cache identity is the opaque native initial prepared message-content hash.
Add no configuration, schema, model, session, implementation, or dependency
dimensions. Store only final accepted codec artifacts or materialized/assessed
plans, not responses, exchanges, proposals, provenance envelopes, or historical
usage. Hits perform ordinary decoding/source-usability checks without new
semantic assessment. Policy/settings changes may still hit; explicit
bypass/refresh controls re-execution. Deterministic work is not cached through
manufactured requests.
Fallback and affected dependent work skip lookup and storage; unaffected accepted
predecessors remain reusable. Refresh replaces an entry only after new acceptance.
Failure preserves an old entry. Cache I/O is best effort and cannot fail otherwise
successful work. Legacy caches remain unread and unmodified, with no checkpoint
resume, canonical source-only plan store, or automatic migration.
Deployments own cache access policy. Writers request conventional maximum modes
and let umask, setgid inheritance, and default ACLs select effective access.
Retain confined paths, symlink rejection, atomic publication, validation, and
recovery without chmod/chown/chgrp convergence or cache migration.
[ADR-0017](../adr/0017-defer-cache-access-policy-to-deployment.md)'s boundary now
applies to the unified cache; output and debug retain separate contracts.
Preserve intrinsic domain payload schemas and logical paths, including supported
fallbacks. Consumers need no index, manifest, persisted receipt, or reporting
sidecar. Publish artifacts and finish capture before a native completion report.
Disclose fallback use independently of diagnostic verbosity. Delivery failure may
leave partial stdout and valid published artifacts; retain those artifacts,
useful locations, and distinct operation/delivery causes without model retries.
Exact report fields and outcomes belong in [CLI Reference](../cli.md).
One native execution group records actual initial, continuation, and nested
validator executions once. Preserve identity, terminal coverage, and
unknown-versus-zero usage without synthetic IDs, copied totals, or historical
execution claims on cache hits.
Debug is never a cache input. Capture requires explicit enablement and a nonempty
effective destination; directory configuration alone does not enable it. Requested
capture is required work: allocate before engine construction/preflight, record
incrementally, and finalize once before successful reporting. Capture failures
fail the invocation while retaining available evidence and primary causes.
Update the relative `latest` link at session start, never completion.
Debug reuses prepared details and native evidence without another preparation/
model call or parallel conversation ledger. Representations are private
diagnostics, not stable serialization contracts or exhaustive provider transcripts.
Use ConfigKit secret-aware inspection and preserve native credential omissions.
Do not directly serialize marked secrets, unrelated environment, or unselected
filesystem content. Do not heuristically scan or rewrite source, prompts,
responses, or arbitrary application text. Selected evidence may contain full
source/reference data, responses, and exchanges; it is not a sanitized bundle.
Operators own access and retention as for other application data.
Writes are atomic where practical. Write, move, overwrite, and deletion targets
must be narrow and explicit. Never automatically delete output or requested debug
bundles. Cache cleanup remains explicit and recoverable.
[Operations](../operations.md) owns physical layout, provisioning,
retention, and recovery guidance.
## Platform And Distribution
Linux is the supported deployment platform. macOS is supported only as a
best-effort development and compilation environment, while Windows is
unsupported. Notarius distributes source releases only: an immutable source
tag and its checked-in release note identify a release. The project does not
publish executable binaries, archives, installers, container images,
checksums, signatures, or package-manager entries. [Release Policy](release.md)
owns publication requirements and completion; maintainer commands and guards
belong to [Source Releases](../release.md).
## Architectural Non-Goals
Notarius does not aim to provide:
- an arbitrary workflow graph or general workflow language;
- source-format or extraction-domain behavior in generic framework packages;
- provider-specific transport contracts exposed to modules;
- structural pipeline composition through ad hoc CLI flags;
- implicit cross-stage behavior that bypasses the fixed pipeline;
- replacement kit APIs, duplicated schedulers, or compatibility parsers;
- a public debug serialization format or mandatory execution ledger;
- abstractions introduced solely for hypothetical future complexity.