Document the new configuration and processing workflow
This commit is contained in:
237
docs/config.md
237
docs/config.md
@@ -1,192 +1,81 @@
|
||||
# Audita Configuration
|
||||
|
||||
This reference describes the version-2 configuration loader in `internal/core/config`.
|
||||
Use the maintained [local](../examples/minimal-config.yml) and
|
||||
[service](../examples/production-config.yml) examples with their separately
|
||||
selected [local profiles](../examples/profiles/local/) and
|
||||
[service profiles](../examples/profiles/service/), plus the optional
|
||||
[testing overlay](../examples/overlays/testing.yml).
|
||||
The [ConfigKit inheritance example](../examples/configkit-inheritance.yml)
|
||||
shows a local child backend reusing and overriding sibling settings.
|
||||
Choose real models for your endpoint; a testing profile makes no cost or quality
|
||||
guarantee. See the [migration guide](migrations/v2.md) for retired inputs.
|
||||
Audita uses version 3 application configuration. See the [migration guide](migrations/v3.md)
|
||||
and [examples](../examples/) for complete projects. Command syntax is owned by
|
||||
[the CLI reference](cli.md).
|
||||
|
||||
## Loading and precedence
|
||||
## Loading and composition
|
||||
|
||||
The application selects one main YAML file: an explicit `--config`, then
|
||||
`/usr/local/etc/audita/config.yml` if present, then `/etc/audita/config.yml`
|
||||
if present. Missing system candidates are skipped; an explicit missing or empty
|
||||
path, inaccessible selected file, or absence of any selected file is an error.
|
||||
`config validate` requires an explicit file. ConfigKit loads built-in defaults,
|
||||
the selected file, an optional named overlay from `overlays/<name>.yml` beside the
|
||||
main file, and only explicitly supplied setting overrides, in that order. Imports
|
||||
and secret-directory bootstrap are disabled. Unknown fields and missing, null,
|
||||
or non-integer `version: 2` are errors. Version-1 files require migration; no
|
||||
compatibility loader or implicit environment configuration remains.
|
||||
An explicit `--config` selects one main YAML file. Without it, ConfigKit searches
|
||||
the working directory, user configuration directory, and Unix system directories
|
||||
for `audita` configuration. If no file is found, built-in defaults apply. An
|
||||
explicit invalid selection or failure of a selected file is an error; search does
|
||||
not continue after a selected file fails. An omitted version uses the v3 schema;
|
||||
a supplied version must be `3`.
|
||||
|
||||
After overlay and explicit CLI override precedence is applied, ConfigKit resolves
|
||||
native sibling inheritance before typed decoding. Any input mapping may use
|
||||
`extends: <sibling-name>`; the child keeps its key, inherits parent settings, and
|
||||
overrides them with its own settings. Selected overlay and CLI values therefore
|
||||
participate in inheritance at their normal precedence. The generic syntax and
|
||||
resolution rules are defined by ConfigKit v0.7.1's
|
||||
[sibling inheritance contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.7.1/docs/config.md#sibling-inheritance).
|
||||
|
||||
`extends` is reserved structural input in base files, imports, overlays, and
|
||||
other application input mappings, including mapping values supplied by explicit
|
||||
CLI setting overrides. ConfigKit consumes it before typed decoding. A previously
|
||||
valid backend whose ID is literally `extends` cannot be represented under this
|
||||
rule. Rename that backend to a non-conflicting ID and update every PromptKit
|
||||
profile that selects it. This compatibility impact means the upgrade does not
|
||||
promise that every previously valid configuration remains valid.
|
||||
|
||||
Inherited values count as explicit input, including zero, false, empty, and
|
||||
null. Existing field rules still apply: for example, inherited backend
|
||||
concurrency must be positive, while an inherited queue capacity of zero is
|
||||
valid. Parent backends undergo the same decoding and application validation as
|
||||
other backends, whether or not a selected profile uses them. A child
|
||||
`credential: null` removes the child's inherited credential; it does not remove
|
||||
the parent's credential reference, which remains subject to ordinary secret
|
||||
resolution.
|
||||
|
||||
Validation runs after overlays, explicit overrides, and derived settings have
|
||||
resolved. An override can therefore repair an invalid lower-precedence setting
|
||||
when that setting can be decoded. ConfigKit retains field origins and derivations
|
||||
for effective-configuration inspection. Its effective YAML redacts resolved
|
||||
credentials.
|
||||
Operator sources may use ConfigKit imports. The selected main file provides the
|
||||
base for an optional `overlays/<name>.yml`, optional `glossary.yml`, and recursive
|
||||
`glossary/` YAML files. Without a main source, automatic companion files are not
|
||||
loaded from the working directory. ConfigKit composes sources, resolves native
|
||||
inheritance, applies explicit overrides, records derived defaults, and validates
|
||||
the result. Its [configuration reference](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.2.0/docs/config.md)
|
||||
owns native composition and provenance semantics.
|
||||
|
||||
### PromptKit profiles and operator prompts
|
||||
|
||||
`profiles_dir` selects a PromptKit configuration root, resolved against the
|
||||
selected main file's directory unless it is absolute. The root may contain
|
||||
keyed `profiles`, `backends`, and `prompts` definitions; it is not limited to
|
||||
profile-only files. IDs come from literal YAML mapping keys, never filenames.
|
||||
Multiple definitions may share a document, and a document may contribute to any
|
||||
of the supported mappings. Directory discovery is recursive, but discovery
|
||||
order does not establish field precedence. Contributions within each source
|
||||
group must compose without conflicting values.
|
||||
PromptKit loads its default catalog and Audita's embedded prompts, followed by
|
||||
optional `prompts.yml` and recursive `prompts/` YAML files beside the application
|
||||
main. Native imports, resource collections, inheritance, and replacement apply.
|
||||
Profiles, backends, models, credentials, admission, and provider parameters belong
|
||||
to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/formats.md).
|
||||
The engine captures its sources for an invocation. Edit files and run a new
|
||||
invocation to pick up changes. Inspection does not require model credentials.
|
||||
|
||||
PromptKit assembles sources in this order: its selected catalogs and owned
|
||||
defaults, Audita's embedded prompt/schema inputs, operator documents under
|
||||
`profiles_dir`, then the application backend mapping. The application mapping
|
||||
owns each configured endpoint, concurrency limit, queue capacity, and credential
|
||||
environment lookup. Other supported provider fields can be contributed by the
|
||||
operator documents. Imports remain disabled. The current maintained examples
|
||||
select separate roots: [local](../examples/profiles/local/) and
|
||||
[service](../examples/profiles/service/), so definitions for one backend do not
|
||||
enter the other example's final configuration.
|
||||
## Pipeline and glossary selections
|
||||
|
||||
Both selected roles must resolve to an existing profile, registered backend,
|
||||
nonblank effective model, and nonblank endpoint. The normalized effective
|
||||
endpoint must match the application backend endpoint. A selected profile cannot
|
||||
introduce a competing `api_key_env` lookup; the application backend credential
|
||||
supplies the runtime key when required. Other unselected profiles may be
|
||||
partial, but every final backend, profile, and prompt definition is validated
|
||||
when the engine is constructed. A malformed or conflicting unused definition
|
||||
can therefore make the selected root invalid. A missing selected root is an
|
||||
operator configuration error; an empty root may contribute no definitions.
|
||||
`glossary_collections` maps names to collections with an ordered `entries` list.
|
||||
Each entry has an `id`, `name`, and `category`; `description`, `aliases`, and
|
||||
`plural` provide optional context. Entries are identified within their collection
|
||||
by `id`. Root `glossaries` and `protected_terms` select collection names. If root
|
||||
protection is omitted, it derives from the root glossary selection.
|
||||
|
||||
Profile IDs and parent IDs are exact, literal keys. PromptKit profiles can use
|
||||
native `extends` inheritance; child fields override inherited fields, while
|
||||
unmentioned controls survive changes to model or backend. For example:
|
||||
`pipeline.stages` is an ordered list of `module` records; supported modules are
|
||||
`glossary`, `homophones`, `spoken_word`, and `grammar`. The default sequence is
|
||||
`glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`. Repeated modules
|
||||
run separately; `[]` runs normalization without model processing. An optional
|
||||
nonempty, unique `id` identifies a record in evidence. Each record may override
|
||||
`glossaries`, `protected_terms`, `confidence_threshold`, and fields under `llm`,
|
||||
`validation`, and `context`. Omitted fields inherit their resolved root values.
|
||||
Explicit empty lists and null list selections clear a selection.
|
||||
|
||||
```yaml
|
||||
profiles:
|
||||
"shared~base":
|
||||
backend: service
|
||||
model: example-model
|
||||
temperature: 0.4
|
||||
top_p: 0.8
|
||||
"proposal fast":
|
||||
extends: "shared~base"
|
||||
model: faster-model
|
||||
temperature: 0
|
||||
```
|
||||
|
||||
Presence matters. Explicit `temperature: 0` is sent as zero; omission leaves it
|
||||
unspecified; null suppresses an inherited temperature. Other fields have their
|
||||
own native null and omission rules. Provider `extra_params` maps merge
|
||||
recursively, and a null provider value is sent as JSON null. See the pinned
|
||||
[PromptKit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.14.0/docs/formats.md)
|
||||
for the complete field semantics.
|
||||
|
||||
Operator documents can also override an embedded prompt/version, including its
|
||||
messages and definition-level output settings. Shared variables supplied by the
|
||||
adapter are rendered only when an effective message references them; Audita does
|
||||
not append omitted shared text. Audita's per-call schemas, structural repair
|
||||
budget, and downstream domain validators remain in force. The engine captures
|
||||
effective definitions and message-file contents when constructed, so edits are
|
||||
observed only by a newly constructed runtime.
|
||||
|
||||
### Migrating PromptKit profile documents
|
||||
|
||||
For the PromptKit v0.11 format, wrap each profile's fields under
|
||||
`profiles: {<literal-id>: ...}`, remove inner `id`, and replace
|
||||
`base_profile` with `extends`. Profile and prompt IDs are mapping keys, so they
|
||||
need not match source filenames. Review zeros, omitted values, and nulls against
|
||||
the native presence rules before deploying. Select a root whose final
|
||||
definitions have valid backend references; eager validation applies even to
|
||||
unselected definitions. See the complete maintained [local](../examples/minimal-config.yml)
|
||||
and [service](../examples/production-config.yml) examples and the pinned
|
||||
[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.14.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api).
|
||||
This incompatible profile-format migration is intended for an Audita major
|
||||
release; no automatic file conversion is provided.
|
||||
|
||||
The [CLI inspection views](cli.md#streams-and-exit-status) show selected
|
||||
controls and field provenance separately from ConfigKit's redacted settings
|
||||
YAML. Inspection resolves both roles without provider calls.
|
||||
Each entry receives its selected context and independent protection. Glossary
|
||||
correction exempts active recognized spellings from protection for that entry;
|
||||
other protected spellings remain protected. A glossary correction must have at
|
||||
least one selected entry at processing preflight. Empty definitions and empty
|
||||
protection remain valid configuration.
|
||||
|
||||
## Settings
|
||||
|
||||
| Field | Default | Rule |
|
||||
| Setting | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `profiles_dir` | `profiles` | Nonempty PromptKit document root; resolved relative to the selected main file, or absolute. |
|
||||
| `backends.<id>.endpoint` | None | Required nonempty endpoint for each backend. At least one backend is required. |
|
||||
| `backends.<id>.concurrency` | `1` | Positive finite capacity; shared by both roles using that backend. |
|
||||
| `backends.<id>.queue_capacity` | `1024` | Nonnegative; zero disables waiting admission. |
|
||||
| `backends.<id>.credential` | Unset | Optional environment-variable secret reference; omission or null means unauthenticated use. |
|
||||
| `llm.proposal_profile` | None | Required nonempty profile ID. |
|
||||
| `llm.validation_profile` | Final proposal profile | Omission inherits the final proposal selection; explicit empty or null fails. |
|
||||
| `llm.transient_retries` | `2` | Nonnegative additional attempts per generation. |
|
||||
| `llm.repair_attempts` | `2` | Additional structural repair generations, from zero through three. |
|
||||
| `llm.operation_timeout` | `15m` | Positive Go duration string per logical call. |
|
||||
| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module keys; repeats are meaningful and an explicit empty list runs no correction modules. |
|
||||
| `output.schema` | `bare-segments` | `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md). |
|
||||
| `llm.proposal_profile` | unset | PromptKit profile for proposals; required for processing entries. |
|
||||
| `llm.validation_profile` | effective proposal profile | Profile for model validation; null clears a selection before fallback. |
|
||||
| `llm.transient_retries` | `2` | Additional transient attempts per generation. |
|
||||
| `llm.repair_attempts` | `2` | Structural repairs, from zero through three. |
|
||||
| `llm.operation_timeout` | `15m` | Positive duration per logical operation. |
|
||||
| `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. |
|
||||
| `chunking.target_sections` | Unset | Optional positive target; omission or null selects automatic chunking. |
|
||||
| `chunking.max_section_tokens` | `8192` | Positive maximum estimate. |
|
||||
| `chunking.min_section_tokens` | `2048` | Positive and no greater than maximum. |
|
||||
| `normalization.max_segment_gap` | `4.0` | Nonnegative numeric seconds. |
|
||||
| `normalization.ellipsis_gap` | `3.5` | Nonnegative numeric seconds. |
|
||||
| `normalization.max_segment_duration` | `60.0` | Positive numeric seconds. |
|
||||
| `normalization.max_segment_tokens` | `2048` | Positive token estimate. |
|
||||
| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. |
|
||||
| `context.description` | Empty | Trimmed prompt context, at most 500 bytes. |
|
||||
| `reporting.enabled` | `true` | Write the ordinary three-file report bundle when processing. |
|
||||
| `reporting.stdout_dir` | `.` | Report directory for stdout transcript output; a `--report-dir` flag can select reports for either transcript destination. |
|
||||
| `diagnostics.debug` | `false` | Persist detailed debug artifacts when processing. |
|
||||
| `diagnostics.work_dir` | `/tmp/audita` | Parent of the retained debug run directory; required only when debug is enabled. |
|
||||
| `thresholds.<module>` | `0.8` | Root confidence threshold for each module, in `[0, 1]`. |
|
||||
| `context.description` | empty | Prompt context; at most 500 bytes after trimming. |
|
||||
| `chunking.target_sections` | unset | Optional positive target. |
|
||||
| `chunking.max_section_tokens`, `chunking.min_section_tokens` | `8192`, `2048` | Positive section bounds; minimum must not exceed maximum. |
|
||||
| `normalization.max_segment_gap`, `normalization.ellipsis_gap` | `4`, `3.5` | Nonnegative seconds. |
|
||||
| `normalization.max_segment_duration`, `normalization.max_segment_tokens` | `60`, `2048` | Positive seconds and token estimate. |
|
||||
| `output.schema` | `bare-segments` | Output format; see [output contracts](integrations/output-files.md). |
|
||||
| `reporting.enabled`, `reporting.stdout_dir` | `true`, `.` | Ordinary report capture and default directory for stdout output. |
|
||||
| `diagnostics.work_dir` | `/tmp/audita` | Parent directory for optional debug capture. |
|
||||
|
||||
Module keys select correction modules; validator chains remain built into them.
|
||||
The loader preserves module order and repetitions. Normalization time settings
|
||||
accept numeric seconds only. The operation timeout uses a duration string.
|
||||
`reporting.stdout_dir` must be nonblank when it supplies the active report
|
||||
destination: for stdout output, or for file output when `--report-dir` is
|
||||
explicitly supplied. File output without that flag uses the transcript's parent,
|
||||
so the configured value is inactive. Reporting-disabled runs do not select a
|
||||
report directory. `diagnostics.work_dir` must be nonblank only with debug
|
||||
enabled. These path requirements are checked when choosing process destinations,
|
||||
not during configuration inspection.
|
||||
|
||||
## Credentials
|
||||
|
||||
A backend credential is a ConfigKit secret reference containing an environment
|
||||
variable name, such as `credential: AUDITA_SERVICE_KEY`. Every reference that
|
||||
survives overlay and override resolution must name a set, nonempty variable,
|
||||
including credentials on backends unused by the selected roles. A replacement
|
||||
reference resolves only the replacement; null removes the reference. General
|
||||
`AUDITA_*` and `OPENROUTER_API_KEY` variables have no implicit setting effect.
|
||||
Secret values are available to the runtime through the resolved reference, while
|
||||
ConfigKit's effective output and provenance contain no resolved secret values.
|
||||
|
||||
Current invocation syntax belongs in [CLI](cli.md).
|
||||
Processing profiles are selected explicitly; no profile or glossary is selected
|
||||
automatically. The validation selection falls back after root and per-entry
|
||||
settings are combined. Null does not disable validation. Numeric zero and false
|
||||
retain their field meaning, subject to validation. Inspect recorded origins and
|
||||
default dependencies using the [native commands](cli.md).
|
||||
|
||||
Reference in New Issue
Block a user