diff --git a/README.md b/README.md index 70aad63..aa22f6e 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,34 @@ # Audita -Audita polishes transcript JSON using glossary-aware and LLM-backed correction -modules. It validates proposed corrections and produces corrected text plus -reports for review and subprocess integration. +Audita polishes transcript JSON with glossary-aware and LLM-backed correction +modules. It validates proposed edits and produces corrected text and reports for +review or subprocess integration. ## Quickstart -From a checkout, build with Go 1.26.0 or newer: +Build with Go 1.27.0 or newer: ```sh go build -o ./bin/audita ./cmd/audita ``` -Processing requires a reachable [compatible LLM endpoint](docs/integrations/openai-compatible-llm.md) -and access to a model you select. The [minimal config](examples/minimal-config.yml) -targets an unauthenticated local endpoint; edit its endpoint and -[local profiles](examples/profiles/local/) to match your service and models. The -[service example](examples/production-config.yml) demonstrates a named credential -reference and a shared backend. Set its referenced environment variable before -validating it. The [testing overlay](examples/overlays/testing.yml) selects both -role profiles, but model cost, speed, and quality depend on your provider choices. - -Check configuration and profile selection offline: +Edit the synthetic endpoint and model in [examples/local/prompts.yml](examples/local/prompts.yml) +for a reachable OpenAI-compatible service. From `examples/local`, run the built +executable (adjust the executable path if needed): ```sh -./bin/audita config validate --config examples/minimal-config.yml -./bin/audita config print-effective --config examples/minimal-config.yml --view profiles +../../bin/audita config validate --config config.yml +../../bin/audita process --config config.yml --output corrected.json transcript.json ``` -Run the included small input with the minimal config: - -```sh -./bin/audita process examples/tiny-transcript.json \ - --config examples/minimal-config.yml \ - --glossary examples/tiny-glossary.yaml \ - --output corrected.json -``` - -This makes LLM requests. Explicit setting flags override the selected file and -optional overlay; general environment setting overrides are no longer read. -By default the command writes `errors.yml`, `recoveries.yml`, and `report.md` -beside `corrected.json`. Use `--reporting=false` to suppress those files, or -`--debug` to retain detailed run artifacts under the configured debug work -directory. Model failures affecting some operations can still yield a completed -transcript with applied corrections; the summary and reports identify them. -See the [v2 migration guide](docs/migrations/v2.md) for old configurations. +Processing makes model requests. Reports are written beside the output by default; +recoverable model errors can appear in a completed run. The [example guide](examples/README.md) +includes service, overlay, prompt editing, preview, and shared glossary projects. +See the [v3 migration guide](docs/migrations/v3.md) for the configuration cutover. ## Documentation -- [CLI reference](docs/cli.md) -- [Configuration and maintained examples](docs/config.md) +- [CLI reference](docs/cli.md) and [configuration](docs/config.md) - [Operations and artifact lifecycle](docs/operations.md) - [Transcript and glossary inputs](docs/integrations/transcript-glossary-files.md) - [Output files and report formats](docs/integrations/output-files.md) diff --git a/docs/adr/0002-glossary-configuration-and-preflight.md b/docs/adr/0002-glossary-configuration-and-preflight.md new file mode 100644 index 0000000..5f8e621 --- /dev/null +++ b/docs/adr/0002-glossary-configuration-and-preflight.md @@ -0,0 +1,42 @@ +# ADR 0002: Glossary configuration and processing readiness + +**Status:** Accepted + +**Date:** 2026-11-10 + +## Context + +Operators need to reuse glossary collections across campaigns and select different +context and protected terms for each ordered correction. Configuration inspection +must remain useful without a transcript, model credentials, or a populated glossary. +A processing run must reject an unusable later correction before publishing outputs +or spending resources on earlier corrections. + +## Decision + +ConfigKit composes typed, named glossary collections with the application +configuration. Each pipeline entry selects context and protection independently; +Audita derives consumer views without changing the resolved configuration. For a +glossary correction, active recognized spellings are exempt from that entry's +protection view. Other protected spellings and validators remain effective. + +After configuration resolution, Audita preflight checks the whole pipeline and +coordinates output preparation. It consumes the typed configuration and acquired +inputs, without reloading configuration or calling a provider. The output session +owns preparation, cleanup, and publication. See the [configuration](../config.md) +and [operations](../operations.md) contracts for current behavior. + +## Alternatives considered + +A separately parsed glossary per invocation would duplicate composition and make +per-entry selection and provenance harder to express. Requiring nonempty +collections in configuration validation would prevent inspection and reuse of +valid empty definitions. Preparing outputs or checking each entry only when it +runs could leave artifacts or incur model work before finding a later error. + +## Consequences + +Configuration and runtime views have distinct responsibilities. Inspection can +show configured selections and origins without presenting derived vocabulary as +configuration. Preflight and the output session need a clear handoff, and the +processing pipeline must carry each entry's own effective settings and identity. diff --git a/docs/adr/README.md b/docs/adr/README.md index 50da111..83bc702 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -4,6 +4,8 @@ - [ADR 0001: Adopt ReleaseKit for release management](0001-releasekit.md) — accepted 2026-10-07. +- [ADR 0002: Glossary configuration and processing readiness](0002-glossary-configuration-and-preflight.md) — + accepted 2026-11-10. Use the [documentation policy's ADR lifecycle](../policy/documentation.md#architecture-decision-records) diff --git a/docs/cli.md b/docs/cli.md index fb9e256..81270b8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,116 +1,45 @@ # Audita CLI Reference -The shared version-2 configuration parser and offline profile inspection serve -`process`, `config validate`, and `config print-effective`. -The [configuration reference](config.md) owns fields, defaults, and validation; -the [migration guide](migrations/v2.md) maps retired inputs. +Audita uses CommandKit for routing and the ConfigKit and PromptKit integrations +for their standard inspection and request commands. Run `audita --help` or +command-specific `--help` for the installed command inventory. The adopted +[CommandKit consumer guide](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/f369d6a78253b6e6fcb1e023fb2e9c3af1d20ff0/docs/consumers/guide.md) +describes the standard command grammar, inspection, comparison, preview, +validation, smoke tests, and their provider-specific caveats. Help and version +do not load application or model configuration. -## Commands and help +## Processing ```sh -audita process --glossary [flags] -audita config validate --config [flags] -audita config print-effective [flags] -audita version +audita process [flags] transcript.json ``` -Root invocation, `--help`, `-h`, and command help succeed without loading a config -file or resolving secrets. Unknown commands and flags return exit code `2`. -`process` accepts exactly one transcript path, either before the flags or after -them. With flags first, put the transcript last. Both config commands reject -positional arguments and process-only flags. +`--config` selects a main configuration and `--overlay` selects its overlay. +`--output` writes the transcript to a file; omission selects stdout. The shared +`--debug` flag enables retained debug artifacts, and `--quiet` suppresses optional +messages. Quiet processing requires an output file or `/dev/null`. See +[operations](operations.md) for output and report lifecycle. -All three commands accept `--config ` and `--overlay `. `config validate` -requires an explicit nonempty `--config`; the other commands select the explicit -file, then `/usr/local/etc/audita/config.yml`, then `/etc/audita/config.yml`. -Missing system candidates are skipped; inaccessible selected files are errors. -No file means a configuration error. `AUDITA_CONFIG` is ignored. The selected -overlay is loaded beside the main file, not used to locate profile files. +Explicit process flags override root settings only. `--glossaries` and +`--protected-terms` each accept one JSON array of collection names or `null`; +`[]` clears the selection. Set pipeline order and per-entry overrides in YAML. +Other supplied flags cover profiles, output schema, description, reporting, +chunking, normalization, confidence thresholds, validation budget, and debug +work directory. `audita process --help` lists the available flags. An omitted flag +does not contribute an override. [Configuration](config.md) defines their values +and inheritance. -`audita version` prints `audita ` and a newline to stdout, with no -application configuration or provider setup. It returns `0` on success with -empty stderr. `help version`, `version -h`, and `version --help` print command -usage to stdout and return `0`. Help follows standard flag parsing and can -short-circuit trailing arguments. Other invalid invocations return `2` with a -usage error on stderr. The command uses ReleaseKit's public -`buildinfo.Version()` accessor. It reports an injected release identifier when -present, otherwise the executable's main-module version when available and not -`(devel)`, and `dev` when neither source provides a release identifier. +## Inspection and outcomes -## Setting overrides +The application configuration scope uses ConfigKit's native effective values, +source evidence, explanation, comparison, and redaction. PromptKit supplies its +own configuration and resource scopes and request preparation, validation, and +explicit smoke diagnostics. Inspection does not require a processing-ready +pipeline or contact a provider. Smoke tests can contact providers and are run +only on explicit request. -Only flags actually supplied override file or overlay settings. `--modules ''` -clears the module pipeline, and an explicit numeric zero reaches configuration -validation. The shared setting flags are: - -| Setting | Flags | -| --- | --- | -| Profile selection | `--proposal-profile`, `--validation-profile` | -| Pipeline and output | `--modules`, `--output-schema` | -| Validation budget | `--validation-max-prompt-tokens` | -| Chunking | `--target-sections`, `--max-section-tokens`, `--min-section-tokens` | -| Normalization (numeric seconds) | `--normalize-max-segment-gap`, `--normalize-ellipsis-gap`, `--normalize-max-segment-duration`, `--normalize-max-segment-tokens` | -| Confidence thresholds | `--glossary-confidence-threshold`, `--homophones-confidence-threshold`, `--spoken-word-confidence-threshold`, `--grammar-confidence-threshold` | -| Context | `--transcript-description` | -| Reporting | `--reporting[=true\|false]`, `--report-dir DIR` | -| Debug | `--debug[=true\|false]`, `--work-dir DIR` | - -`process` alone accepts `--glossary` (required) and `--output`. -An omitted or empty output value selects stdout. Nonempty output filenames are -used literally, including whitespace. -Reporting defaults to enabled. `--reporting=false` disables all three ordinary -reports; an explicit false overrides a true file or overlay setting. For stdout -output, reports use the effective `reporting.stdout_dir`, which defaults to the -invocation working directory. For file output, they go beside the transcript -unless `--report-dir` is explicitly supplied. The flag selects the report -directory for either transcript destination; a config-file or overlay value for -`reporting.stdout_dir` alone affects stdout output only. A selected report -directory must be nonblank when reporting is enabled. With reporting disabled, -the report destination is inactive and is not probed. `--debug` separately -enables retained detailed artifacts under `--work-dir`; selecting `--work-dir` -alone does not enable capture. Configuration inspection does not create or -probe these output directories. -Per-role model, base URL, key, timeout, retry, and concurrency flags have been -removed. Models and generation controls belong to PromptKit profiles, backend -credentials and capacity to configuration, and retry/deadline settings to the -shared LLM section. Legacy configuration environment overrides are ignored. - -## Streams and exit status - -`config validate` loads settings and both selected PromptKit profiles offline, -then prints `config is valid` on success. `config print-effective` accepts -`--view settings|profiles|explain` and defaults to `settings`. Settings writes -ConfigKit's redacted effective-settings YAML. Profiles writes an Audita-owned -YAML summary of selected ID, backend, model, sampling controls, per-attempt -timeout, service tier, and reasoning effort for each role. It omits credentials, -endpoints, and arbitrary extra provider parameters. Explain writes a YAML -sequence of field paths, source origins, and recorded derivations without -resolved setting values. An inherited field may also include an `inheritance` -sequence; each step identifies `from`, `to`, and `reference` paths as JSON -pointers. These fields describe the effective inheritance chain without -including configuration values. A direct child override reports its own source -instead. Debug configuration origins use the same explanation shape. These views -share the same settings and profile validation path as `process`. Unknown views -are usage errors. The effective settings YAML is for inspection and is not -guaranteed reloadable. - -For `process`, `--output` writes the corrected transcript to a file and leaves -stdout empty on success. Without it, stdout carries transcript JSON only on -success. Enabled reporting creates `errors.yml`, `recoveries.yml`, and -`report.md` in the selected directory. Successful stderr begins with -`Completed: N corrections applied; E work units with errors; R recoveries.` -Report and debug locations follow only when created. Model-operation failures -can coexist with a completed run and applied corrections; read the YAML reports -for details. Debug artifacts are created only with `--debug`. See the -[report contract](integrations/output-files.md). - -Exit codes are `0` for success, `1` for runtime or output failure, and `2` for -usage or configuration input errors. Invalid settings and profiles return `2`; -broken embedded assets, caller cancellation, fatal processing, and required -output failures return `1`. A late report-write failure can follow a published -transcript; exit status and report status remain authoritative. -Configuration commands inspect profiles without contacting a provider. - -See the [input files](integrations/transcript-glossary-files.md), -[subprocess contract](integrations/subprocess.md), and -[artifact lifecycle](operations.md) for their respective details. +Process stdout contains transcript JSON when no output file is selected. Ordinary +reports may record recoverable model failures even when processing completes. +Exit status `0` means success, `1` runtime or output failure, `2` invalid usage +or input, and native comparison may return `3` for differences. See the +[output contract](integrations/output-files.md) and [subprocess contract](integrations/subprocess.md). diff --git a/docs/config.md b/docs/config.md index 69fdd78..0c1e031 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,192 +1,81 @@ # Audita Configuration -This reference describes the version-2 configuration loader in `internal/core/config`. -Use the maintained [local](../examples/minimal-config.yml) and -[service](../examples/production-config.yml) examples with their separately -selected [local profiles](../examples/profiles/local/) and -[service profiles](../examples/profiles/service/), plus the optional -[testing overlay](../examples/overlays/testing.yml). -The [ConfigKit inheritance example](../examples/configkit-inheritance.yml) -shows a local child backend reusing and overriding sibling settings. -Choose real models for your endpoint; a testing profile makes no cost or quality -guarantee. See the [migration guide](migrations/v2.md) for retired inputs. +Audita uses version 3 application configuration. See the [migration guide](migrations/v3.md) +and [examples](../examples/) for complete projects. Command syntax is owned by +[the CLI reference](cli.md). -## Loading and precedence +## Loading and composition -The application selects one main YAML file: an explicit `--config`, then -`/usr/local/etc/audita/config.yml` if present, then `/etc/audita/config.yml` -if present. Missing system candidates are skipped; an explicit missing or empty -path, inaccessible selected file, or absence of any selected file is an error. -`config validate` requires an explicit file. ConfigKit loads built-in defaults, -the selected file, an optional named overlay from `overlays/.yml` beside the -main file, and only explicitly supplied setting overrides, in that order. Imports -and secret-directory bootstrap are disabled. Unknown fields and missing, null, -or non-integer `version: 2` are errors. Version-1 files require migration; no -compatibility loader or implicit environment configuration remains. +An explicit `--config` selects one main YAML file. Without it, ConfigKit searches +the working directory, user configuration directory, and Unix system directories +for `audita` configuration. If no file is found, built-in defaults apply. An +explicit invalid selection or failure of a selected file is an error; search does +not continue after a selected file fails. An omitted version uses the v3 schema; +a supplied version must be `3`. -After overlay and explicit CLI override precedence is applied, ConfigKit resolves -native sibling inheritance before typed decoding. Any input mapping may use -`extends: `; the child keeps its key, inherits parent settings, and -overrides them with its own settings. Selected overlay and CLI values therefore -participate in inheritance at their normal precedence. The generic syntax and -resolution rules are defined by ConfigKit v0.7.1's -[sibling inheritance contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.7.1/docs/config.md#sibling-inheritance). - -`extends` is reserved structural input in base files, imports, overlays, and -other application input mappings, including mapping values supplied by explicit -CLI setting overrides. ConfigKit consumes it before typed decoding. A previously -valid backend whose ID is literally `extends` cannot be represented under this -rule. Rename that backend to a non-conflicting ID and update every PromptKit -profile that selects it. This compatibility impact means the upgrade does not -promise that every previously valid configuration remains valid. - -Inherited values count as explicit input, including zero, false, empty, and -null. Existing field rules still apply: for example, inherited backend -concurrency must be positive, while an inherited queue capacity of zero is -valid. Parent backends undergo the same decoding and application validation as -other backends, whether or not a selected profile uses them. A child -`credential: null` removes the child's inherited credential; it does not remove -the parent's credential reference, which remains subject to ordinary secret -resolution. - -Validation runs after overlays, explicit overrides, and derived settings have -resolved. An override can therefore repair an invalid lower-precedence setting -when that setting can be decoded. ConfigKit retains field origins and derivations -for effective-configuration inspection. Its effective YAML redacts resolved -credentials. +Operator sources may use ConfigKit imports. The selected main file provides the +base for an optional `overlays/.yml`, optional `glossary.yml`, and recursive +`glossary/` YAML files. Without a main source, automatic companion files are not +loaded from the working directory. ConfigKit composes sources, resolves native +inheritance, applies explicit overrides, records derived defaults, and validates +the result. Its [configuration reference](https://forge.maximumdirect.net/base-layer/configkit/src/tag/v1.2.0/docs/config.md) +owns native composition and provenance semantics. ### PromptKit profiles and operator prompts -`profiles_dir` selects a PromptKit configuration root, resolved against the -selected main file's directory unless it is absolute. The root may contain -keyed `profiles`, `backends`, and `prompts` definitions; it is not limited to -profile-only files. IDs come from literal YAML mapping keys, never filenames. -Multiple definitions may share a document, and a document may contribute to any -of the supported mappings. Directory discovery is recursive, but discovery -order does not establish field precedence. Contributions within each source -group must compose without conflicting values. +PromptKit loads its default catalog and Audita's embedded prompts, followed by +optional `prompts.yml` and recursive `prompts/` YAML files beside the application +main. Native imports, resource collections, inheritance, and replacement apply. +Profiles, backends, models, credentials, admission, and provider parameters belong +to PromptKit configuration; see its [format reference](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/formats.md). +The engine captures its sources for an invocation. Edit files and run a new +invocation to pick up changes. Inspection does not require model credentials. -PromptKit assembles sources in this order: its selected catalogs and owned -defaults, Audita's embedded prompt/schema inputs, operator documents under -`profiles_dir`, then the application backend mapping. The application mapping -owns each configured endpoint, concurrency limit, queue capacity, and credential -environment lookup. Other supported provider fields can be contributed by the -operator documents. Imports remain disabled. The current maintained examples -select separate roots: [local](../examples/profiles/local/) and -[service](../examples/profiles/service/), so definitions for one backend do not -enter the other example's final configuration. +## Pipeline and glossary selections -Both selected roles must resolve to an existing profile, registered backend, -nonblank effective model, and nonblank endpoint. The normalized effective -endpoint must match the application backend endpoint. A selected profile cannot -introduce a competing `api_key_env` lookup; the application backend credential -supplies the runtime key when required. Other unselected profiles may be -partial, but every final backend, profile, and prompt definition is validated -when the engine is constructed. A malformed or conflicting unused definition -can therefore make the selected root invalid. A missing selected root is an -operator configuration error; an empty root may contribute no definitions. +`glossary_collections` maps names to collections with an ordered `entries` list. +Each entry has an `id`, `name`, and `category`; `description`, `aliases`, and +`plural` provide optional context. Entries are identified within their collection +by `id`. Root `glossaries` and `protected_terms` select collection names. If root +protection is omitted, it derives from the root glossary selection. -Profile IDs and parent IDs are exact, literal keys. PromptKit profiles can use -native `extends` inheritance; child fields override inherited fields, while -unmentioned controls survive changes to model or backend. For example: +`pipeline.stages` is an ordered list of `module` records; supported modules are +`glossary`, `homophones`, `spoken_word`, and `grammar`. The default sequence is +`glossary`, `homophones`, `glossary`, `spoken_word`, `grammar`. Repeated modules +run separately; `[]` runs normalization without model processing. An optional +nonempty, unique `id` identifies a record in evidence. Each record may override +`glossaries`, `protected_terms`, `confidence_threshold`, and fields under `llm`, +`validation`, and `context`. Omitted fields inherit their resolved root values. +Explicit empty lists and null list selections clear a selection. -```yaml -profiles: - "shared~base": - backend: service - model: example-model - temperature: 0.4 - top_p: 0.8 - "proposal fast": - extends: "shared~base" - model: faster-model - temperature: 0 -``` - -Presence matters. Explicit `temperature: 0` is sent as zero; omission leaves it -unspecified; null suppresses an inherited temperature. Other fields have their -own native null and omission rules. Provider `extra_params` maps merge -recursively, and a null provider value is sent as JSON null. See the pinned -[PromptKit format reference](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.14.0/docs/formats.md) -for the complete field semantics. - -Operator documents can also override an embedded prompt/version, including its -messages and definition-level output settings. Shared variables supplied by the -adapter are rendered only when an effective message references them; Audita does -not append omitted shared text. Audita's per-call schemas, structural repair -budget, and downstream domain validators remain in force. The engine captures -effective definitions and message-file contents when constructed, so edits are -observed only by a newly constructed runtime. - -### Migrating PromptKit profile documents - -For the PromptKit v0.11 format, wrap each profile's fields under -`profiles: {: ...}`, remove inner `id`, and replace -`base_profile` with `extends`. Profile and prompt IDs are mapping keys, so they -need not match source filenames. Review zeros, omitted values, and nulls against -the native presence rules before deploying. Select a root whose final -definitions have valid backend references; eager validation applies even to -unselected definitions. See the complete maintained [local](../examples/minimal-config.yml) -and [service](../examples/production-config.yml) examples and the pinned -[consumer migration guide](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.14.0/docs/consumers/pkg-promptkit.md#migrate-from-the-previous-configuration-api). -This incompatible profile-format migration is intended for an Audita major -release; no automatic file conversion is provided. - -The [CLI inspection views](cli.md#streams-and-exit-status) show selected -controls and field provenance separately from ConfigKit's redacted settings -YAML. Inspection resolves both roles without provider calls. +Each entry receives its selected context and independent protection. Glossary +correction exempts active recognized spellings from protection for that entry; +other protected spellings remain protected. A glossary correction must have at +least one selected entry at processing preflight. Empty definitions and empty +protection remain valid configuration. ## Settings -| Field | Default | Rule | +| Setting | Default | Meaning | | --- | --- | --- | -| `profiles_dir` | `profiles` | Nonempty PromptKit document root; resolved relative to the selected main file, or absolute. | -| `backends..endpoint` | None | Required nonempty endpoint for each backend. At least one backend is required. | -| `backends..concurrency` | `1` | Positive finite capacity; shared by both roles using that backend. | -| `backends..queue_capacity` | `1024` | Nonnegative; zero disables waiting admission. | -| `backends..credential` | Unset | Optional environment-variable secret reference; omission or null means unauthenticated use. | -| `llm.proposal_profile` | None | Required nonempty profile ID. | -| `llm.validation_profile` | Final proposal profile | Omission inherits the final proposal selection; explicit empty or null fails. | -| `llm.transient_retries` | `2` | Nonnegative additional attempts per generation. | -| `llm.repair_attempts` | `2` | Additional structural repair generations, from zero through three. | -| `llm.operation_timeout` | `15m` | Positive Go duration string per logical call. | -| `pipeline.modules` | `[glossary, homophones, glossary, spoken_word, grammar]` | Ordered module keys; repeats are meaningful and an explicit empty list runs no correction modules. | -| `output.schema` | `bare-segments` | `bare-segments` or `audita-v1`. See [output formats](integrations/output-files.md). | +| `llm.proposal_profile` | unset | PromptKit profile for proposals; required for processing entries. | +| `llm.validation_profile` | effective proposal profile | Profile for model validation; null clears a selection before fallback. | +| `llm.transient_retries` | `2` | Additional transient attempts per generation. | +| `llm.repair_attempts` | `2` | Structural repairs, from zero through three. | +| `llm.operation_timeout` | `15m` | Positive duration per logical operation. | | `validation.max_prompt_tokens` | `2048` | Positive validation batching budget. | -| `chunking.target_sections` | Unset | Optional positive target; omission or null selects automatic chunking. | -| `chunking.max_section_tokens` | `8192` | Positive maximum estimate. | -| `chunking.min_section_tokens` | `2048` | Positive and no greater than maximum. | -| `normalization.max_segment_gap` | `4.0` | Nonnegative numeric seconds. | -| `normalization.ellipsis_gap` | `3.5` | Nonnegative numeric seconds. | -| `normalization.max_segment_duration` | `60.0` | Positive numeric seconds. | -| `normalization.max_segment_tokens` | `2048` | Positive token estimate. | -| `thresholds.glossary`, `.homophones`, `.spoken_word`, `.grammar` | `0.8` each | Confidence thresholds in `[0, 1]`. | -| `context.description` | Empty | Trimmed prompt context, at most 500 bytes. | -| `reporting.enabled` | `true` | Write the ordinary three-file report bundle when processing. | -| `reporting.stdout_dir` | `.` | Report directory for stdout transcript output; a `--report-dir` flag can select reports for either transcript destination. | -| `diagnostics.debug` | `false` | Persist detailed debug artifacts when processing. | -| `diagnostics.work_dir` | `/tmp/audita` | Parent of the retained debug run directory; required only when debug is enabled. | +| `thresholds.` | `0.8` | Root confidence threshold for each module, in `[0, 1]`. | +| `context.description` | empty | Prompt context; at most 500 bytes after trimming. | +| `chunking.target_sections` | unset | Optional positive target. | +| `chunking.max_section_tokens`, `chunking.min_section_tokens` | `8192`, `2048` | Positive section bounds; minimum must not exceed maximum. | +| `normalization.max_segment_gap`, `normalization.ellipsis_gap` | `4`, `3.5` | Nonnegative seconds. | +| `normalization.max_segment_duration`, `normalization.max_segment_tokens` | `60`, `2048` | Positive seconds and token estimate. | +| `output.schema` | `bare-segments` | Output format; see [output contracts](integrations/output-files.md). | +| `reporting.enabled`, `reporting.stdout_dir` | `true`, `.` | Ordinary report capture and default directory for stdout output. | +| `diagnostics.work_dir` | `/tmp/audita` | Parent directory for optional debug capture. | -Module keys select correction modules; validator chains remain built into them. -The loader preserves module order and repetitions. Normalization time settings -accept numeric seconds only. The operation timeout uses a duration string. -`reporting.stdout_dir` must be nonblank when it supplies the active report -destination: for stdout output, or for file output when `--report-dir` is -explicitly supplied. File output without that flag uses the transcript's parent, -so the configured value is inactive. Reporting-disabled runs do not select a -report directory. `diagnostics.work_dir` must be nonblank only with debug -enabled. These path requirements are checked when choosing process destinations, -not during configuration inspection. - -## Credentials - -A backend credential is a ConfigKit secret reference containing an environment -variable name, such as `credential: AUDITA_SERVICE_KEY`. Every reference that -survives overlay and override resolution must name a set, nonempty variable, -including credentials on backends unused by the selected roles. A replacement -reference resolves only the replacement; null removes the reference. General -`AUDITA_*` and `OPENROUTER_API_KEY` variables have no implicit setting effect. -Secret values are available to the runtime through the resolved reference, while -ConfigKit's effective output and provenance contain no resolved secret values. - -Current invocation syntax belongs in [CLI](cli.md). +Processing profiles are selected explicitly; no profile or glossary is selected +automatically. The validation selection falls back after root and per-entry +settings are combined. Null does not disable validation. Numeric zero and false +retain their field meaning, subject to validation. Inspect recorded origins and +default dependencies using the [native commands](cli.md). diff --git a/docs/development.md b/docs/development.md index 4225ced..37c6bbf 100644 --- a/docs/development.md +++ b/docs/development.md @@ -41,7 +41,7 @@ ownership. Commit discipline and agent Git restrictions are in the ## Baseline Validation -Use Go 1.26.0 or newer for source builds. The selected release toolchain is +Use Go 1.27.0 or newer for source builds. The selected release toolchain is documented in the [release procedure](release.md#prerequisites). From the repository root: diff --git a/docs/integrations/openai-compatible-llm.md b/docs/integrations/openai-compatible-llm.md index 8121963..27d20c4 100644 --- a/docs/integrations/openai-compatible-llm.md +++ b/docs/integrations/openai-compatible-llm.md @@ -1,26 +1,26 @@ # OpenAI-Compatible LLM Integration -Audita uses the pinned [PromptKit](https://gitea.maximumdirect.net/eric/promptkit) +Audita uses the pinned [PromptKit](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09) built-in OpenAI-compatible transport. PromptKit owns chat-completions wire framing, provider envelope parsing, transient retries, per-generation timeouts, and JSON Schema repair. A backend's configured base endpoint must support the selected profile's model and structured JSON-schema responses; local configuration inspection does not contact the provider. -Audita selects a role profile, supplies its registered backend credential, and -passes the embedded response schema as a complete per-call output contract. It -prepares each call once, checks that the prepared target still matches the -inspected backend and endpoint, then executes that same prepared request. A -profile cannot introduce a separate credential environment lookup. An absent -backend credential permits an unauthenticated local service. +Audita selects role profiles independently for each pipeline entry and supplies +its domain response schema and per-operation budget. PromptKit owns backend and +credential selection, request preparation, transport, retries, repairs, and +admission. Audita executes the same prepared handle and retains its own domain +validation. Profiles may use native endpoint and credential settings; local +inspection does not resolve a credential value or call the provider. `llm.transient_retries` counts additional attempts for eligible transient failures; explicit zero makes one attempt. `llm.repair_attempts` counts additional generations for schema-invalid model output. The operation timeout covers preparation, capacity waiting, retries, generation, and repair. Caller cancellation or an earlier deadline also applies. See [configuration](../config.md#settings) -for defaults and [PromptKit v0.14.0's consumer contract](https://gitea.maximumdirect.net/eric/promptkit/src/tag/v0.14.0/docs/consumers/pkg-promptkit.md) -for the pinned protocol and retry details. +for defaults and the [PromptKit consumer contract](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.21.0/docs/consumers/pkg-promptkit.md) +for protocol and retry details. A model operation that exhausts structural repair, a live-parent local operation timeout, capacity rejection, transport failure, malformed provider response, @@ -31,4 +31,4 @@ continue. Caller cancellation, broken assets or configuration, and unknown internal failures remain fatal and cancel peers. Audita decodes into domain types only after PromptKit validates the JSON. Ordinary reports use safe reason codes and counts rather than provider messages. Full request/response content -is written only with `--debug`; see [operations](../operations.md#diagnostics-directory). +is written only with `--debug`; see [operations](../operations.md#debug-evidence). diff --git a/docs/integrations/output-files.md b/docs/integrations/output-files.md index c99a54a..a5d2423 100644 --- a/docs/integrations/output-files.md +++ b/docs/integrations/output-files.md @@ -5,7 +5,7 @@ The corrected transcript uses the selected `output.schema`: `bare-segments` or `audita-v1`. Its schema and text rules are unchanged by resilience reporting. See [configuration](../config.md#settings) for selection and -[operations](../operations.md#required-output-files) for placement and writes. +[operations](../operations.md#outputs-and-lifecycle) for placement and writes. With reporting enabled, Audita produces three ordinary files: @@ -46,7 +46,7 @@ remain in opt-in debug files. ## Error and Recovery Records An `errors` record with `scope: work_unit` identifies zero-based `stage_index` -and `chunk_index`, module key/instance, initial segment range, applied-correction +and `chunk_index`, optional `stage_id`, module key/instance, initial segment range, applied-correction count, and a `failures` sequence. Each failure identifies the operation (`proposal`, `validation`, or `validation_input`), optional validator and zero-based batch index, safe reason/message, sorted proposal indices when @@ -59,7 +59,7 @@ For `validation_decisions_invalid`, proposal indices identify only candidates whose required decisions were unavailable. Other candidates in the same batch may still be approved or rejected normally. -Each recovery identifies its work unit and successful operation, with sorted +Each recovery identifies its work unit, optional `stage_id`, and successful operation, with sorted `reasons`, a safe message, available execution counts, and sorted `filtered_proposals` details. One recovery can contain `output_repaired`, `service_recovered`, and `proposals_filtered` together. Filtering covers @@ -79,8 +79,8 @@ is an error even when unexpected indices were also ignored. `report.md` includes run ID, status, corrections, work coverage, error and recovery totals, input/output paths, and full work-unit error and recovery matrices. Rows show every initial chunk with 1-based chunk number and initial -segment range; columns follow configured module-instance order, including -repeats. Completed error cells are `0` or `1`; recovery cells are operation +segment range; columns follow configured pipeline order, including repeated modules. Labels show +the one-based position and module, plus the operator ID when supplied. Completed error cells are `0` or `1`; recovery cells are operation counts. `—` means not started and `n*` means interrupted with n facts recorded. An empty stage or chunk set is identified as having no work units. Short reason breakdowns count records containing each reason; a recovery can have several reasons, so those @@ -91,10 +91,9 @@ files. It does not include transcript excerpts or correction lists. With `--debug`, `correction-ledger.json` contains proposed candidates and their application, rejection, or interruption details. `llm-execution.json` -holds logical calls and PromptKit execution events; `utilization-diagnostics.json` -groups backend capacity and timing observations while preserving known and -complete usage semantics. Interaction request/response artifacts, selected -profiles, configuration provenance, and input snapshots are also debug-only. +holds logical calls and PromptKit execution events. Interaction request/response +artifacts and input snapshots are also debug-only. Records carry zero-based +`stage_index` and optional `stage_id` where an entry is identified. These files are diagnostic evidence, not an alternate ordinary report schema. Each candidate-backed ledger entry includes `proposed_corrected_text`, the literal replacement span, even when it is `""` for a deletion. Applied entries diff --git a/docs/integrations/subprocess.md b/docs/integrations/subprocess.md index 47c1a4f..4796d37 100644 --- a/docs/integrations/subprocess.md +++ b/docs/integrations/subprocess.md @@ -6,15 +6,15 @@ The [CLI reference](../cli.md) owns command syntax, streams, and exit semantics; ## Invocation -Use an argument array with the transcript path and required glossary argument, -following the [process examples](../cli.md#commands-and-help). For machine workflows, +Use an argument array with the transcript path and configured glossary selections, +following the [process examples](../cli.md#processing). For machine workflows, select an explicit transcript file destination and a configuration path. Reports go beside the transcript unless an explicit `--report-dir` selects another directory. For stdout output, reports use the effective `reporting.stdout_dir` (defaulting to the invocation working directory), including a `--report-dir` override. A config-file or overlay value for `reporting.stdout_dir` alone does not redirect file-output reports. `--work-dir` selects a debug root, while -configuration or `--debug` enables capture; the path flag alone does not. +`--debug` enables capture; the path flag alone does not. Supplying a configuration path also makes file selection predictable; only explicitly referenced credential variables are read from the environment. @@ -26,7 +26,7 @@ or cancellation in the parent process when needed. ## Result Handling Interpret the exit result according to the -[CLI output and exit contract](../cli.md#streams-and-exit-status). +[CLI output and exit contract](../cli.md#inspection-and-outcomes). Consume transcript/report data through the [output-file contract](output-files.md), and treat stderr as human-readable detail. The three reports may be absent after early configuration or destination @@ -38,7 +38,7 @@ transcript may have been written before a later required report failure, so the file alone does not prove completion. If processing fails, preserve available evidence and follow -[operations recovery](../operations.md#failure-inspection-and-recovery). +[operations recovery](../operations.md#recovery). Ordinary reports are independent of optional retained debug evidence. Parent cancellation reaches the child through its context; Audita joins workers and attempts failed-status report finalization before exiting. Uncatchable @@ -49,4 +49,4 @@ file nor a debug directory alone establishes successful completion. - [Transcript and glossary inputs](transcript-glossary-files.md) - [LLM endpoint compatibility](openai-compatible-llm.md) -- [Configuration inspection commands](../cli.md#commands-and-help) +- [Configuration inspection commands](../cli.md#inspection-and-outcomes) diff --git a/docs/integrations/transcript-glossary-files.md b/docs/integrations/transcript-glossary-files.md index 2af4be9..204b646 100644 --- a/docs/integrations/transcript-glossary-files.md +++ b/docs/integrations/transcript-glossary-files.md @@ -28,23 +28,11 @@ The [tiny transcript](../../examples/tiny-transcript.json) is a complete copyabl input. Invalid JSON, unsupported top-level shapes, empty transcripts, invalid field types, duplicate supplied IDs, and the validation failures above are rejected. -## Glossary YAML Contract +## Glossary collections -The top-level object contains a `glossary` list with at least one entry. - -| Entry field | Type and validation | -| --- | --- | -| `name` | Non-empty string | -| `category` | Non-empty string | -| `summary` | Non-empty string | -| `aliases` | Optional list of non-empty strings | -| `plural` | Optional string | - -The parser checks empty strings without trimming them. Unknown YAML fields are -ignored by the glossary parser, unlike the stricter -[configuration-file schema](../config.md#loading-and-precedence). - -The [tiny glossary](../../examples/tiny-glossary.yaml) is a complete copyable input. -Malformed YAML, an empty/missing glossary, invalid shapes, missing required text, -and empty aliases are rejected. These formats describe input acceptance, not -additional vocabulary restrictions. +Glossary definitions are composed as application configuration, including the +optional companion inputs described by the [configuration reference](../config.md). +The [local example](../../examples/local/glossary.yml) provides a complete +collection. Configuration may contain empty collections; processing preflight +requires every glossary-correction entry to select at least one glossary entry. +The previous standalone `glossary` file format is not a processing input. diff --git a/docs/internal/cli-config.md b/docs/internal/cli-config.md index daa246a..62a6f75 100644 --- a/docs/internal/cli-config.md +++ b/docs/internal/cli-config.md @@ -9,7 +9,7 @@ CLI wiring, and maintained examples. Definitions of behavior belong to the 1. Update fields, defaults, loading, and validation in `internal/core/config`. 2. Update file, selected overlay, credential reference, and CLI mapping as applicable. Preserve the precedence defined in the - [configuration contract](../config.md#loading-and-precedence), including the + [configuration contract](../config.md#loading-and-composition), including the distinct validation and effective-config commands. 3. Update redaction when secrets are involved. 4. Review configuration tests and related CLI tests for precedence, validation, @@ -29,33 +29,18 @@ CLI wiring, and maintained examples. Definitions of behavior belong to the The `version` route uses ReleaseKit's public `buildinfo.Version()` accessor and stays independent of application runtime assembly. Preserve its behavior from -the [CLI contract](../cli.md#commands-and-help) when changing the command; the +the [CLI contract](../cli.md) when changing the command; the adoption decision is recorded in [ADR 0001](../adr/0001-releasekit.md). ## Validating Examples Keep complete copyable files in [examples](../../examples/) and link to them from their canonical contract documentation. Keep them valid and secret-free. -From the repository root, validate the maintained configuration examples: - -```sh -go run ./cmd/audita config validate --config examples/minimal-config.yml -go run ./cmd/audita config validate --config examples/configkit-inheritance.yml -AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml -AUDITA_SERVICE_KEY=synthetic-example go run ./cmd/audita config validate --config examples/production-config.yml --overlay testing -go run ./cmd/audita config print-effective --config examples/minimal-config.yml --view profiles -go run ./cmd/audita config print-effective --config examples/configkit-inheritance.yml --view explain -``` - -The local and ConfigKit inheritance files select `profiles/local`; the service -file selects `profiles/service`, including the service testing profiles. Keep -these roots separate because PromptKit validates every final definition in a -selected root, including definitions that are not selected for a role. - -Only referenced credentials read environment values; use a controlled synthetic -value when validating the service example. Check affected CLI examples against the -parser and use schema tests for maintained transcript/glossary inputs. Example -validation should not invoke a live LLM merely to check documentation. +Run the commands in the [example guide](../../examples/README.md) from each +project's directory. `internal/cli/examples_integration_test.go` exercises the +maintained projects with a controlled provider, and the preview fixture can be +prepared without inference. Review automatic companion sources and overlays in +their actual project directories. Keep service credentials synthetic in tests. ## Runtime Change Checks diff --git a/docs/internal/diagnostics-reporting.md b/docs/internal/diagnostics-reporting.md index 8019e3c..cd0bccf 100644 --- a/docs/internal/diagnostics-reporting.md +++ b/docs/internal/diagnostics-reporting.md @@ -24,36 +24,27 @@ retain their execution counts and never produce a recovery record. `internal/core/diagnostics.OutputSession` owns active destination checks, transcript preflight, report initialization, regular-file replacement, and direct stream publication. Transcript input is protected from early report writes but -may be the final transcript destination. The LLM runtime records successful reads -through PromptKit's profile filesystem and supplies those source paths to CLI; +may be the final transcript destination. The LLM runtime records consumed PromptKit sources and effective host message +files and supplies those source paths to CLI; no second filename-based discovery rule is used. -CLI composes it after configuration/profile inspection and before reading -inputs or dispatching model work. It does not create ordinary report parents. +CLI composes it after configuration, transcript acquisition, and preflight +readiness and before dispatching model work. It does not create ordinary report parents. On a final write error it attempts other enabled reports and one failed-status pass; a published transcript is never erased by a later report failure. -Debug is independent. Only `diagnostics.debug` creates a unique retained child +Debug is independent. The shared `--debug` flag enables a unique retained child under `diagnostics.work_dir`. CLI passes empty diagnostics paths and nil writers -when debug is off. With debug on, source/normalized snapshots, invocation, -effective settings, origins, profiles, initial chunk summary, correction ledger, -indexed execution/utilization, and model interactions are written. Runner -candidate capture includes proposals later rejected or interrupted. Ledger -mapping uses optional string pointers so an empty proposed replacement is -serialized for every candidate and an empty applied replacement appears only -when that candidate applied; see the [public artifact contract](../integrations/output-files.md#debug-only-files). -Debug write failures reach the CLI through an explicit concurrency-safe diagnostic -callback and produce redacted stderr messages without aborting processing. -Completed validator decisions returned with a fatal batch error remain available -to the debug ledger; they never authorize applying an incompletely validated edit. -Runner decision records carry the validator's explicit approved, rejected, or -unavailable disposition into the ledger. Unavailable candidates have a distinct -top-level ledger disposition, while fatal interruptions remain interrupted when -no rejection or unavailable decision was returned. Successful validation -interaction payloads contain the full decoded response, final content, and -reconciliation outcomes; ignored indices do not become ledger candidates. -Every debug writer uses all configured backend secrets for redaction, including -unselected backend credentials and source snapshots. Debug can still contain -sensitive transcript and prompt content. +when debug is off. With debug enabled, it captures source and normalized +snapshots, invocation, effective settings, correction ledger, LLM execution +observations, and model interactions. Candidate capture includes proposals later +rejected or interrupted. Ledger mapping preserves empty proposed and applied +replacements and distinguishes unavailable candidates from interrupted work. + +A concurrency-safe diagnostic callback reports optional write failures without +aborting processing. Diagnostic redaction uses credential source metadata; debug +artifacts can still contain sensitive transcript and prompt content. Execution +position and optional operator ID are carried separately from generated safe +artifact path coordinates. ## Key Tests diff --git a/docs/internal/llm-runtime.md b/docs/internal/llm-runtime.md index 1dc70c6..d222a3f 100644 --- a/docs/internal/llm-runtime.md +++ b/docs/internal/llm-runtime.md @@ -1,60 +1,22 @@ # Audita LLM Runtime -## Client Boundary +`llm.NewRuntime` assembles one captured PromptKit engine using its default +catalog, Audita's embedded prompts and schemas, and optional operator sources +anchored to the recorded application main context. Captured host sources and +message files are passed to output collision checks. An embedded-only diagnostic +construction distinguishes shipped asset failures from operator-source failures; +it never replaces the requested engine. -`llm.NewRuntime` assembles one PromptKit engine for both roles. PromptKit's -catalogs and defaults are followed by Audita's embedded prompt and schema -inputs, operator documents from `profiles_dir`, and the highest-priority -application backend mapping. Audita owns backend endpoints, capacity settings, -and resolved credentials; native PromptKit layering owns profile inheritance, -prompt overrides, and provider-specific fields. Imports are disabled. The -runtime inspects both selected roles offline, requiring complete model and -endpoint values and rejecting competing profile credential lookups. A -role-bound `llm.RoleClient` implements `structuredoutput.Client`; proposal and -validator contracts alias that interface. Both roles share engine-local backend -capacity. +Each resolved pipeline entry receives role-bound clients with its selected +profiles and effective budgets. Clients share engine-local backend admission. +The adapter uses `PrepareExecution` and `RunPrepared` on the same handle, passes +Audita's per-call schema and repair budget, and decodes only validated results. +PromptKit owns backend and credential selection. The adapter uses captured +credential source names for artifact redaction without inspecting unrelated +environment values. See the [integration contract](../integrations/openai-compatible-llm.md) +for failure behavior and the [configuration reference](../config.md) for sources. -Successful construction captures the final definitions and referenced message -contents. The runtime does not reload operator documents or message files during -calls. If final engine construction fails, an embedded-only diagnostic -construction distinguishes shipped asset failures from invalid operator input. -An asset failure remains an asset error; when embedded construction succeeds, -the original failure remains a configuration error. The diagnostic engine -never replaces or executes the requested configuration, and caller cancellation -skips this path. The captured inspection also supplies an internal source -ledger: all loaded host YAML plus effective, source-relative host -`content_file` resources are tracked for output-collision checks. - -For each logical call, the adapter starts an operation context and a diagnostic -record before preparation. It supplies the reserved hardening text, the selected -profile and resolved backend credential, and the complete embedded schema/repair -contract. `PrepareExecution` produces the exact messages and hashes retained in -an Audita snapshot. The adapter checks the prepared backend and endpoint before -`RunPrepared`. It records available final content, usage, model, and profile -metadata and decodes only a validated result. Call hooks expose logical -start/finish records and PromptKit execution events for debug utilization. - -The [integration contract](../integrations/openai-compatible-llm.md) describes -wire ownership and failure behavior. `structuredoutput.Failure` has a safe -message, fixed reason code, typed cause, and recoverable/fatal/canceled class. -Counts distinguish provider attempts, transient retries, and structural -repairs, including on errors. Secrets from all configured backends, including unselected -ones, are available to the diagnostics redaction boundary. - -CLI explain output and debug configuration origins share a value-free projection -of effective field origins. The [CLI contract](../cli.md) defines its optional -inheritance evidence fields. - -## Orchestration - -CLI processing supplies two role clients from the same engine to the runner. -PromptKit's per-backend admission applies to both roles and every generation, -retry, and repair. The runner still processes module instances in order, -collects concurrent section work deterministically, and cancels remaining -section work only on fatal failures. `llm.DiagnosticsWriter` remains the protected -request/response artifact writer. `llm.ObservationCollector` captures metadata -events under a short mutex and freezes them after runner work joins. Its logical -records include preparation failures; summary usage comes from terminal events -once per call and retains knownness and completeness for each field. Runner wall -and validator timings remain separate from provider attempts. Ordinary reports -derive compact runner outcomes; full observations are debug-only. +The observation collector tracks logical calls and execution events. Ordinary +reports use compact safe outcomes; request and response evidence is optional +debug material. The runner preserves ordered pipeline traversal and deterministic +section collection while canceling and joining peers on fatal errors. diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 9f3d846..2346b5a 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -88,4 +88,4 @@ outcomes are defined by the [report contract](../integrations/output-files.md). - `internal/modules/*/module_test.go` - `internal/framework/modules/registry_test.go` - `internal/framework/proposal_generation/*_test.go` -- `internal/cli/run_test.go` (pipeline/report integration) +- `internal/cli/process_ordered_integration_test.go` (pipeline/report integration) diff --git a/docs/internal/output-schemas.md b/docs/internal/output-schemas.md index 9cfa7b8..319ea04 100644 --- a/docs/internal/output-schemas.md +++ b/docs/internal/output-schemas.md @@ -12,7 +12,7 @@ selectable values and defaults belong to Configuration validation checks registry support. CLI orchestration resolves the selected encoder, passes the final transcript to it, and routes the encoded bytes -to the destination defined by the [CLI contract](../cli.md#streams-and-exit-status). +to the destination defined by the [CLI contract](../cli.md#inspection-and-outcomes). An unsupported key fails validation or resolution rather than selecting an implicit fallback. diff --git a/docs/internal/overview.md b/docs/internal/overview.md index ae5ac12..ae3a88d 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -14,6 +14,7 @@ Executable and command orchestration: Core deterministic components: - `internal/core/config` +- `internal/core/glossary` - `internal/core/schema` - `internal/core/normalization` - `internal/core/chunking` @@ -24,6 +25,7 @@ Core deterministic components: Framework orchestration and contracts: - `internal/framework/contracts` +- `internal/framework/preflight` - `internal/framework/modules` - `internal/framework/proposal_generation` - `internal/framework/proposals` @@ -66,7 +68,7 @@ artifact lifecycle. ## Component Responsibilities Core components implement configuration defaults/loading/validation/redaction, -transcript and glossary schemas, normalization and chunking, output encoding, +transcript schemas and derived glossary views, normalization and chunking, output encoding, diagnostics artifact helpers, and public report structures. Framework components implement module and validator contracts, proposal diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 099a871..fedc00b 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -4,8 +4,9 @@ The [input contract](../integrations/transcript-glossary-files.md) owns input formats; [configuration](../config.md) owns settings; the [output contract](../integrations/output-files.md) owns serialized results. -1. CLI filesystem adapters read transcript and glossary files. Schema parsing - produces source objects; transcript segments enter canonical normalization. +1. CLI loads resolved configuration and transcript input. Preflight checks all + selected glossary views and roles before the output session prepares resources. + Transcript segments then enter canonical normalization. 2. Normalization sorts and merges segments using configured time and token limits, reassigning IDs where needed. 3. The runner partitions the normalized transcript exactly once before its diff --git a/docs/internal/validators.md b/docs/internal/validators.md index 0a2daba..edaae24 100644 --- a/docs/internal/validators.md +++ b/docs/internal/validators.md @@ -116,19 +116,20 @@ is covered by [report assembly](diagnostics-reporting.md). - `internal/validators/*_test.go` - `internal/framework/validators/*_test.go` - `internal/framework/processreport/correction_ledger_test.go` -- `internal/cli/run_test.go` +- `internal/cli/process_ordered_integration_test.go` ## Supplemental Validation Context -Request construction prepares one segment-position lookup and protected vocabulary -for all candidates. Glossary selection remains candidate-specific, including the -proposed replacement text. +Request construction prepares one segment-position lookup and a matcher from the +selected context entries for all candidates. Context selection remains candidate-specific, +including the proposed replacement text. The independent protection view is used +by the deterministic guard and application check. Each candidate includes the immediately preceding and following source segments available within its fixed chunk, labeled `preceding` or `following` with speaker, text, and categories, plus glossary entries whose names, aliases, or recognized plurals appear in its original/resulting text or those neighbors. -Glossary matching reuses the protected vocabulary; entries retain glossary order. +Glossary matching uses the same recognition rules; entries retain selection order. Model-facing candidate and neighboring context omit internal segment IDs. This supplies evidence such as a troupe definition for a proposed `Jesters` correction or an adjacent speaker turn explaining deliberate repetition. A correction diff --git a/docs/migrations/v2.md b/docs/migrations/v2.md index da97cdf..16b0ec8 100644 --- a/docs/migrations/v2.md +++ b/docs/migrations/v2.md @@ -2,8 +2,8 @@ Audita v2 adopts ConfigKit for configuration and PromptKit for LLM execution. Version-1 configuration files, environment overlays, and retired flags are not -accepted. Copy a maintained [local](../../examples/minimal-config.yml) or -[service](../../examples/production-config.yml) example, then select real models +accepted. Copy a maintained [local](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.1.1/examples/minimal-config.yml) or +[service](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.1.1/examples/production-config.yml) example, then select real models in the accompanying `examples/profiles/` files. A testing overlay is shown at `examples/overlays/testing.yml`; its model choice carries no cost, speed, or quality guarantee. @@ -122,7 +122,7 @@ capacity and usage fields retain known/complete flags. Debug runs write `effective-config.redacted.yaml`, `selected-profiles.yaml`, `config-origins.yaml`, `utilization-diagnostics.json`, and `llm-execution.json`. Request and response artifacts remain redacted debugging -material. See [operations](../operations.md#diagnostics-directory) for placement +material. See [operations](../operations.md#debug-evidence) for placement and paths, [configuration](../config.md) for all settings, and the [CLI reference](../cli.md) for current flags. diff --git a/docs/migrations/v3.md b/docs/migrations/v3.md new file mode 100644 index 0000000..1ccb158 --- /dev/null +++ b/docs/migrations/v3.md @@ -0,0 +1,23 @@ +# Migrating to application configuration v3 + +Audita v3 is a hard cutover. Move application settings to `version: 3` and use +[the configuration reference](../config.md) and [maintained examples](../../examples/) +to construct the new files. The previous configuration and standalone glossary +formats are not loaded by the new processing path. + +- Replace `pipeline.modules` with ordered `pipeline.stages` records containing + `module` and any per-entry overrides. An explicit empty list runs normalization. +- Define named `glossary_collections` in configuration or its companion imports; + select them using `glossaries` and `protected_terms`. Remove `--glossary` and + `--modules` from invocations. +- Move profiles and backends out of the application configuration and retired + `profiles_dir` layout into PromptKit resources. Use its current list records, + IDs, credential sources, and backend admission settings. Select proposal and + validation profiles in application `llm` settings. +- Use the shared `--debug` flag for capture; `diagnostics.work_dir` remains the + parent directory. Review report and output destinations before running. + +Inspect configuration and resources before processing, then try a controlled +small transcript. Processing preflight checks every glossary correction and +selected role before preparing outputs or invoking a model. See [operations](../operations.md) +for lifecycle and recovery. diff --git a/docs/operations.md b/docs/operations.md index 9f5431e..7081437 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,94 +1,58 @@ # Audita Operations -This guide owns physical output placement and failure handling. The -[CLI reference](cli.md) owns flags and streams, [configuration](config.md) owns -settings, and the [output contract](integrations/output-files.md) owns schemas. +This guide owns output placement and failure handling. The [CLI reference](cli.md) +owns invocation, [configuration](config.md) owns settings, and the +[output contract](integrations/output-files.md) owns serialized formats. -## Normal Workflow +## Processing -1. Validate a version-2 configuration and its selected profiles with - `audita config validate --config config.yml`. -2. Prepare a transcript and glossary following the [input contract](integrations/transcript-glossary-files.md). -3. Run `audita process input.json --config config.yml --glossary glossary.yaml --output corrected.json`. -4. Check the exit status, stderr summary, corrected transcript, and reports. - A completed run can contain recoverable model errors and still apply - corrections that passed every required validator. +Inspect the application and PromptKit resources, prepare a transcript, then run +`audita process` with the desired destination. The [examples](../examples/README.md) +show complete projects. Processing preflight checks every configured glossary +correction and selected profile before output preparation and model work. Empty +protection and an empty pipeline are valid. A completed run can contain +recoverable model errors while still applying independently validated edits. -## Required Output Files +## Outputs and lifecycle -Reporting is enabled by default. For file transcript output, `errors.yml`, -`recoveries.yml`, and `report.md` go in the transcript output's parent directory -unless `--report-dir` is explicitly supplied. For stdout transcript output, they -go in the effective `reporting.stdout_dir` (default `.`), including a -`--report-dir` override. A config-file or overlay value for -`reporting.stdout_dir` alone does not redirect file-output reports. An active -selected report directory must be nonblank. `--reporting=false` suppresses all -three reports and leaves their destination inactive, so it is not probed merely -because `--report-dir` was supplied. It does not suppress the stderr summary or -fatal errors. Stdout contains only transcript JSON; file output leaves it empty. +Reporting is enabled by default. For transcript file output, `errors.yml`, +`recoveries.yml`, and `report.md` are placed beside that file unless `--report-dir` +is supplied. For stdout transcript output they use the effective report directory. +Disabling reporting leaves its destination inactive and unprobed. Quiet mode +requires a transcript file or `/dev/null`; it suppresses optional messages, not +failures. Stdout carries transcript JSON when selected. -Existing report directories and parents for regular transcript files must be -writable. Audita prevents report writes from overwriting transcript, glossary, -loaded application configuration, host YAML discovered under `profiles_dir`, -or host message files referenced by effective PromptKit `content_file` fields, -and rejects conflicting output destinations. Source paths come from the -runtime's captured configuration; unrelated files in the selected directory -remain eligible output destinations. The final transcript may intentionally -replace its own input after successful processing; other input/configuration/ -profile sources remain protected. -Preparation never truncates an existing transcript. Device and pipe destinations -(such as `/dev/null` and named pipes) receive ordinary writes at publication time; -they are not opened during preflight or replaced with regular files. A pipe write -can wait for its reader, and stream write errors are fatal. With reporting enabled, -reports still use the output path's parent unless an explicit `--report-dir` -selects another directory; disable reporting when that placement is unsuitable -for a stream destination. +Before processing, Audita checks destination collisions against the transcript, +consumed application and PromptKit sources, and effective host message files. An +intentional in-place transcript replacement is allowed. A regular transcript +output is prepared in a sibling temporary file, leaving the final destination +untouched until successful publication. Pipes and devices are opened only for +delivery; a pipe can wait for its reader. Parent directories for required outputs +must already exist. -With reporting enabled, all three reports are initialized as `in_progress` before -input processing. Output is encoded fully before publication. Each regular output -file is replaced through a temporary file on its destination filesystem: -transcript first, then `errors.yml`, -`recoveries.yml`, and `report.md`. This is per-file replacement, not a -multi-file transaction. New files honor the process umask; replacing an existing -file preserves its permission bits. A late report error may leave the transcript -published; the process exits nonzero and attempts failed-status reports. On fatal processing -or cancellation, no completed transcript is published, but partial reports are -finalized where possible. An uncatchable termination can leave `in_progress`. +Enabled reports are initialized as `in_progress` before processing. The +transcript is published before final reports; the files do not form a single +transaction. A late report failure may therefore leave a published transcript. +Fatal errors and cancellation clean unpublished temporary resources and attempt +failed-status final reports. An uncatchable termination can leave `in_progress` +evidence. Read exit status and reports together rather than treating a file's +existence as proof of success. -## Diagnostics Directory +## Debug evidence -`--debug` or `diagnostics.debug` enables detailed evidence independently of -reporting. One directory named `run-` is created under -`diagnostics.work_dir` (default `/tmp/audita`), which `--work-dir` can override, -and retained on success or failure. Selecting `--work-dir` alone does not enable -capture. Without debug, no directory or payload files are created. The parent -is created when needed only for debug. Failure to -write optional debug evidence prints a redacted stderr diagnostic but does not -change correction decisions or the exit status. +The shared `--debug` flag enables retained artifacts in a run directory beneath +`diagnostics.work_dir`; selecting a work directory alone does not enable capture. +Optional debug preparation or writes can produce redacted warnings without +changing correction decisions or the required output result. Debug material can +include transcript content, prompts, responses, and correction decisions. It is +not anonymized; protect it according to the input's sensitivity. Configured +credential values are redacted at the diagnostic boundary. -Debug evidence includes invocation and effective settings, configuration -origins, selected profiles, redacted source and normalized snapshots, -normalization and initial chunk summaries, all captured proposed candidates -and validator decisions in `correction-ledger.json`, execution/utilization -records, and model interaction files. Individual artifact writes can fail; a -directory's existence does not prove every file was written. These files can -contain transcript text and prompt/response payloads. Configured backend -credentials are redacted, but the transcript itself is not anonymized. Protect -and retain the directory according to the sensitivity of the input. +## Recovery -## Failure Inspection and Recovery - -Exit `0` means processing and every enabled required output completed. Exit -`1` means cancellation, fatal processing, or required-output failure. Exit `2` -identifies usage or configuration/destination selection errors. Recoverable -model-operation failures appear in `errors.yml` as one record per affected -work unit; they do not automatically make the run fail. `recoveries.yml` -records successful operations that needed retry, repair, or proposal filtering. -`report.md` shows every initial chunk and configured module instance, including -interrupted and unattempted work. Ordinary reports omit transcript excerpts and -full decision ledgers; enable debug when those are needed. - -Read stderr and report status first. If a required output failed after a -transcript was published, do not treat the file alone as proof of a completed -run. Fix the input, service, or destination problem and rerun. Audita has no -resume/checkpoint API. +Exit `0` indicates completion of processing and required outputs; nonzero status +indicates an input, processing, cancellation, or required-output failure. +Recoverable model-operation failures and successful repairs or retries are +recorded separately in the ordinary YAML reports. The Markdown matrices identify +pipeline position, module, and optional operator ID. Fix the input, service, or +destination and rerun when necessary; Audita has no resume or checkpoint API. diff --git a/docs/oss/attribution.md b/docs/oss/attribution.md index c6da857..7cba9c9 100644 --- a/docs/oss/attribution.md +++ b/docs/oss/attribution.md @@ -29,7 +29,7 @@ License and notices: [CC BY-SA 4.0](licenses/CC-BY-SA-4.0.txt) | ## ConfigKit -Selected module: `gitea.maximumdirect.net/eric/ConfigKit` at `v0.7.1`. +Selected module: `go.maximumdirect.net/configkit` at `v1.2.0`. Copyright (C) 2026 Eric Rakestraw @@ -37,14 +37,17 @@ SPDX-License-Identifier: LGPL-3.0-or-later License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | [GPL v3](licenses/GPL-3.0.txt) | -[COPYRIGHT](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/6fffde4af8f31f99d3ee295c426eb3234cb0fb77/COPYRIGHT). +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e/COPYRIGHT). -[Source](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/6fffde4af8f31f99d3ee295c426eb3234cb0fb77). +[Source](https://forge.maximumdirect.net/base-layer/configkit/src/commit/c1c7e87f2cdf77a0cbf3327831ac05578e756b9e). ## ReleaseKit Selected module: `go.maximumdirect.net/releasekit` at `v0.9.0`. +Material: ReleaseKit runtime and adapted launcher and tag-delivery recipes in +`scripts/release.sh` and `.woodpecker/release.yml`. + Copyright (C) 2026 Eric Rakestraw SPDX-License-Identifier: LGPL-3.0-or-later @@ -55,6 +58,48 @@ License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | [Source](https://forge.maximumdirect.net/base-layer/releasekit/src/commit/2180aa868ec2caa95fc8279fd59d5f0ebac41bd4). +## CommandKit + +Selected module: `go.maximumdirect.net/commandkit` at `v0.6.0`. + +Copyright (C) 2026 Eric Rakestraw + +SPDX-License-Identifier: LGPL-3.0-or-later + +License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | +[GPL v3](licenses/GPL-3.0.txt) | +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/1613ff95cab60581de5d86f6e9e2a3cc091a44a0/COPYRIGHT). + +[Source](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/1613ff95cab60581de5d86f6e9e2a3cc091a44a0). + +## CommandKit ckconfig + +Selected module: `go.maximumdirect.net/ckconfig` at `v0.7.0`. + +Copyright (C) 2026 Eric Rakestraw + +SPDX-License-Identifier: LGPL-3.0-or-later + +License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | +[GPL v3](licenses/GPL-3.0.txt) | +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/64f6453b8453f914f272d427bb4ae6f59deacd2b/integrations/ckconfig/COPYRIGHT). + +[Source](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/64f6453b8453f914f272d427bb4ae6f59deacd2b). + +## CommandKit ckprompt + +Selected module: `go.maximumdirect.net/ckprompt` at `v0.7.0`. + +Copyright (C) 2026 Eric Rakestraw + +SPDX-License-Identifier: LGPL-3.0-or-later + +License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | +[GPL v3](licenses/GPL-3.0.txt) | +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/f369d6a78253b6e6fcb1e023fb2e9c3af1d20ff0/integrations/ckprompt/COPYRIGHT). + +[Source](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/f369d6a78253b6e6fcb1e023fb2e9c3af1d20ff0). + ## Go runtime and standard library Selected toolchain: Go 1.27.3 for the distributed Linux binaries. @@ -64,10 +109,10 @@ Copyright 2009 The Go Authors. SPDX-License-Identifier: BSD-3-Clause License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | -[LICENSE](https://github.com/golang/go/blob/go1.27.3/LICENSE) | -[PATENTS](https://github.com/golang/go/blob/go1.27.3/PATENTS). +[LICENSE](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/LICENSE) | +[PATENTS](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/PATENTS). -[Source](https://github.com/golang/go/commit/go1.27.3). +[Source](https://github.com/golang/go/commit/2aae1000ea1d4b28923b96db28acf4e3b99d6aff). ## Go vendored golang.org/x/crypto @@ -78,10 +123,10 @@ Copyright 2009 The Go Authors. SPDX-License-Identifier: BSD-3-Clause License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | -[LICENSE](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/crypto/LICENSE) | -[PATENTS](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/crypto/PATENTS). +[LICENSE](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/crypto/LICENSE) | +[PATENTS](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/crypto/PATENTS). -[Source](https://github.com/golang/go/commit/go1.27.3). +[Source](https://github.com/golang/go/commit/2aae1000ea1d4b28923b96db28acf4e3b99d6aff). ## Go vendored golang.org/x/net @@ -92,10 +137,10 @@ Copyright 2009 The Go Authors. SPDX-License-Identifier: BSD-3-Clause License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | -[LICENSE](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/net/LICENSE) | -[PATENTS](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/net/PATENTS). +[LICENSE](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/net/LICENSE) | +[PATENTS](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/net/PATENTS). -[Source](https://github.com/golang/go/commit/go1.27.3). +[Source](https://github.com/golang/go/commit/2aae1000ea1d4b28923b96db28acf4e3b99d6aff). ## Go vendored golang.org/x/sys @@ -106,10 +151,10 @@ Copyright 2009 The Go Authors. SPDX-License-Identifier: BSD-3-Clause License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | -[LICENSE](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/sys/LICENSE) | -[PATENTS](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/sys/PATENTS). +[LICENSE](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/sys/LICENSE) | +[PATENTS](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/sys/PATENTS). -[Source](https://github.com/golang/go/commit/go1.27.3). +[Source](https://github.com/golang/go/commit/2aae1000ea1d4b28923b96db28acf4e3b99d6aff). ## Go vendored golang.org/x/text @@ -120,10 +165,10 @@ Copyright 2009 The Go Authors. SPDX-License-Identifier: BSD-3-Clause License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | -[LICENSE](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/text/LICENSE) | -[PATENTS](https://github.com/golang/go/blob/go1.27.3/src/vendor/golang.org/x/text/PATENTS). +[LICENSE](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/text/LICENSE) | +[PATENTS](https://github.com/golang/go/blob/2aae1000ea1d4b28923b96db28acf4e3b99d6aff/src/vendor/golang.org/x/text/PATENTS). -[Source](https://github.com/golang/go/commit/go1.27.3). +[Source](https://github.com/golang/go/commit/2aae1000ea1d4b28923b96db28acf4e3b99d6aff). ## go-yaml v4 @@ -169,7 +214,7 @@ License and notices: [Apache 2.0](licenses/Apache-2.0.txt). ## PromptKit -Selected module: `gitea.maximumdirect.net/eric/promptkit` at `v0.14.0`. +Selected module: `go.maximumdirect.net/promptkit` at `v0.21.0`. Copyright (C) 2026 Eric Rakestraw @@ -177,37 +222,9 @@ SPDX-License-Identifier: LGPL-3.0-or-later License and notices: [LGPL v3](licenses/LGPL-3.0.txt) | [GPL v3](licenses/GPL-3.0.txt) | -[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit/src/commit/143c6e9b3a675872117b0471e2ff8bcb25055a54/COPYRIGHT). +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/COPYRIGHT). -[Source](https://gitea.maximumdirect.net/eric/promptkit/src/commit/143c6e9b3a675872117b0471e2ff8bcb25055a54). - -## Promptkit OpenRouter Catalog - -Selected module: `gitea.maximumdirect.net/eric/promptkit-backend-openrouter/v3` at `v3.20260930.0`. - -Copyright (C) 2026 Eric Rakestraw - -SPDX-License-Identifier: MIT - -License and notices: [MIT](licenses/MIT-Catalogs.txt) | -[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/f17d208cb7ad1fb043cda2266cd3011d11b40aa0/COPYRIGHT) | -[LICENSE](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/f17d208cb7ad1fb043cda2266cd3011d11b40aa0/LICENSE). - -[Source](https://gitea.maximumdirect.net/eric/promptkit-backend-openrouter/src/commit/f17d208cb7ad1fb043cda2266cd3011d11b40aa0). - -## Promptkit Rakestrawhome Catalog - -Selected module: `gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/v3` at `v3.20260930.0`. - -Copyright (C) 2026 Eric Rakestraw - -SPDX-License-Identifier: MIT - -License and notices: [MIT](licenses/MIT-Catalogs.txt) | -[COPYRIGHT](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/993443d88c451901f4c917fa00c204fd3b54d28d/COPYRIGHT) | -[LICENSE](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/993443d88c451901f4c917fa00c204fd3b54d28d/LICENSE). - -[Source](https://gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome/src/commit/993443d88c451901f4c917fa00c204fd3b54d28d). +[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09). ## yaml.v3 @@ -229,3 +246,31 @@ License and notices: [Apache 2.0](licenses/Apache-2.0.txt) | [NOTICE](https://github.com/go-yaml/yaml/blob/f6f7691b1fdeb513f56608cd2c32c51f8194bf51/NOTICE). [Source](https://github.com/go-yaml/yaml/commit/f6f7691b1fdeb513f56608cd2c32c51f8194bf51). + +## Cobra + +Selected module: `github.com/spf13/cobra` at `v1.10.1`. + +Copyright 2013-2023 The Cobra Authors + +SPDX-License-Identifier: Apache-2.0 + +License and notices: [Apache 2.0](licenses/Apache-2.0.txt) | +[LICENSE](https://github.com/spf13/cobra/blob/7da941c3547e93b8c9f70bbd3befca79c6335388/LICENSE.txt). + +[Source](https://github.com/spf13/cobra/tree/7da941c3547e93b8c9f70bbd3befca79c6335388). + +## pflag + +Selected module: `github.com/spf13/pflag` at `v1.0.9`. + +Copyright (c) 2012 Alex Ogier. All rights reserved. + +Copyright (c) 2012 The Go Authors. All rights reserved. + +SPDX-License-Identifier: BSD-3-Clause + +License and notices: [BSD 3-Clause](licenses/BSD-3-Clause-Go.txt) | +[LICENSE](https://github.com/spf13/pflag/blob/10438578954bba2527fe5cae3684d4532b064bbe/LICENSE). + +[Source](https://github.com/spf13/pflag/tree/10438578954bba2527fe5cae3684d4532b064bbe). diff --git a/docs/oss/licenses/MIT-Catalogs.txt b/docs/oss/licenses/MIT-Catalogs.txt deleted file mode 100644 index ae99260..0000000 --- a/docs/oss/licenses/MIT-Catalogs.txt +++ /dev/null @@ -1,18 +0,0 @@ -MIT License - -Copyright (c) 2026 Eric Rakestraw - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and -associated documentation files (the "Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the -following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial -portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT -LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO -EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE -USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 2b38111..3254e3f 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -77,8 +77,8 @@ hidden global state. - Validate corrections before final mutation, including local proposal checks, decision cardinality, and apply-time safety. Provider-side schema enforcement does not replace these checks. -- Keep LLM concurrency bounded and context-aware through PromptKit's engine-local - per-backend admission. Proposal and validation calls selecting the same backend +- Use PromptKit's context-aware, engine-local per-backend admission for + LLM concurrency. Proposal and validation calls selecting the same backend share that capacity, including retries and repairs. Preserve deterministic collection order. Recoverable model-operation failures stay local to their proposal response or validation batch; fatal invariant/application failure @@ -120,13 +120,19 @@ responsibilities and should remain separate. Implementation details belong in ## Dependencies -Keep Audita dependency-light and prefer the Go standard library for CLI parsing, -filesystems, synchronization, and tests. ConfigKit owns configuration loading, -presence, provenance, and secret resolution; PromptKit owns prompt/profile -loading, OpenAI-compatible transport, retries, repairs, and backend admission. -Audita owns domain meaning, role selection, and deterministic processing. +Keep Audita dependency-light and prefer the Go standard library where no adopted +library owns the boundary. ConfigKit owns configuration composition, presence, +recorded resolution, and provenance. PromptKit owns prompt, profile, backend, +credential, transport, retry, repair, and admission behavior. CommandKit and its +integrations own command routing and their standard inspection and request +surfaces. Audita owns domain meaning, role selection, transcript processing, +and presentation of its processing results. -Do not introduce broad CLI, dependency-injection, workflow, logging, or plugin +Pass resolved typed configuration to preflight and processing. Preflight checks +execution readiness before output preparation or model work; the output session +owns resources, cleanup, and publication. The glossary and readiness decision is +recorded in [ADR 0002](../adr/0002-glossary-configuration-and-preflight.md). +Do not introduce broad dependency-injection, workflow, logging, or plugin frameworks without a concrete implemented need and focused tests. ## Testing and Documentation diff --git a/docs/policy/release.md b/docs/policy/release.md index 304f04f..1fb43c2 100644 --- a/docs/policy/release.md +++ b/docs/policy/release.md @@ -184,6 +184,15 @@ only through an explicit project-specific policy decision explaining the concrete risk and why ordinary change review or automation is insufficient. Do not add routine confirmations for checks the command can perform itself. +## Audita's Selected Candidate Checks + +Audita uses ReleaseKit v0.9.0 with race-enabled offline tests, full Go vet, +formatting, Markdown links, module distribution checks, and the required target +builds. Race checks address shared processing and client state; vet and format +checks catch common defects and mechanical drift; link and module checks protect +the distributed source tree. The [runbook](../release.md) specifies the selected +toolchain, native commands, and asynchronous recovery. + ## Publication Guards Automate the checks needed to publish the intended source under a fresh version: diff --git a/docs/release.md b/docs/release.md index 1e5bdf9..1fa8b47 100644 --- a/docs/release.md +++ b/docs/release.md @@ -80,7 +80,9 @@ bash scripts/release.sh publish VERSION ``` The command runs the offline Go test suite with race detection, full Go vet, -formatting, Markdown link, and module distribution checks. It builds the +formatting, Markdown link, and module distribution checks. Race detection covers +shared processing and client state; vet and formatting catch common Go defects and +mechanical drift; the link and module checks protect the distributed source tree. It builds the profile's Linux amd64 and arm64 targets and checks the native version output. It creates an annotated tag whose message comes from the committed note, then pushes the exact tag object. A successful publication completes the source release without waiting for CI or diff --git a/docs/releases/v3.0.0.md b/docs/releases/v3.0.0.md index bbe365f..11d38c2 100644 --- a/docs/releases/v3.0.0.md +++ b/docs/releases/v3.0.0.md @@ -38,8 +38,8 @@ profile files require migration from the v2 release format. 3. Select a configuration root whose final definitions are valid. PromptKit now validates unused definitions too, including backend references, and conflicting values within a directory source group are errors. The maintained examples - use separate [local](../../examples/profiles/local/) and - [service](../../examples/profiles/service/) roots. Runtime source edits take + use separate [local](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/examples/profiles/local/) and + [service](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/examples/profiles/service/) roots. Runtime source edits take effect on the next invocation. 4. Rename any application backend literally named `extends` and update its profile references. ConfigKit now reserves that key for inheritance. @@ -49,9 +49,9 @@ profile files require migration from the v2 release format. The [configuration reference](../config.md#promptkit-profiles-and-operator-prompts) contains the migrated format and presence rules. Start from the maintained -[local](../../examples/minimal-config.yml), -[service](../../examples/production-config.yml), or -[application inheritance](../../examples/configkit-inheritance.yml) examples. +[local](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/examples/minimal-config.yml), +[service](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/examples/production-config.yml), or +[application inheritance](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/examples/configkit-inheritance.yml) examples. The [CLI reference](../cli.md) describes report selection, and the [release procedure](../release.md) describes source publication and asynchronous Linux binary delivery. diff --git a/docs/roadmap/glossary-stages.md b/docs/roadmap/glossary-stages.md index c227e2d..4a9242e 100644 --- a/docs/roadmap/glossary-stages.md +++ b/docs/roadmap/glossary-stages.md @@ -17,7 +17,7 @@ implementation is pending. The The current [CLI](../cli.md) accepts one glossary input for a processing run. The [pipeline configuration](../config.md) supports repeated glossary stages, but does not select different glossary inputs for each occurrence. Glossary -files currently have a separate [YAML input contract](../integrations/transcript-glossary-files.md#glossary-yaml-contract). +files currently have a separate [YAML input contract](../integrations/transcript-glossary-files.md#glossary-collections). Make glossaries and glossary collections reusable, composable configuration assets loaded and composed by ConfigKit. Allow ordered stages to select different resolved diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 7792405..fd818c5 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -573,6 +573,11 @@ Verify existing partial-success and unavailable-candidate evidence survives. ## Stage 14: Finish the cutover documentation, examples, and attribution +**Completed.** Canonical guidance, the decision record, migration guide, +maintained projects, release procedure, and dependency attribution reflect the +cutover. Controlled-provider project checks and documented offline inspections +passed; historical example links point to the unchanged v3.0.0 tag. + Update canonical owners as behavior lands; use this stage to reconcile them. Follow the [documentation policy](../policy/documentation.md) and glossary [durable record](glossary-stages.md#durable-decision-record). diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..0c5cf1f --- /dev/null +++ b/examples/README.md @@ -0,0 +1,74 @@ +# Maintained examples + +Build Audita from the repository root with Go 1.27 or newer. The commands below +assume the executable is on `PATH`. Each project selects its main file from its +own directory, which also determines companion resources and overlay context. +All models, credentials, names, and endpoints are illustrative. + +## Local processing + +From `examples/local`, edit `prompts.yml` to match a reachable OpenAI-compatible +local service and its model. Then run: + +```sh +audita config validate --config config.yml +audita process --config config.yml --output corrected.json transcript.json +``` + +Processing makes model requests. Reports are written beside the selected output +unless disabled. For an offline validator preview, from the same directory: + +```sh +audita promptkit show prepared audita/validators.meaning_reversal_review \ + --config config.yml --profile local \ + --var "PayloadJSON=$(cat validation-candidate.json)" \ + --var 'TranscriptDescriptionBlock=' +``` + +This renders the supplied candidate without inference. The native command help +and [CLI reference](../docs/cli.md) describe additional inspection and validation. + +## Service and overlay + +From `examples/service`, edit the synthetic endpoint and models in `prompts.yml`. +Set `AUDITA_EXAMPLE_KEY` to your service credential before a real request. Inspect +without a provider call: + +```sh +audita config validate --config config.yml --overlay testing +audita promptkit show profile service-test --config config.yml +``` + +The example does not promise a provider's model availability, price, or quality. + +## Edit a prompt and run again + +From `examples/prompt-edit`, inspect the operator replacement: + +```sh +audita promptkit show prompt audita/modules.grammar.proposal --config config.yml +``` + +Edit its message in `prompts.yml`, repeat the inspection, and run `audita process +--config config.yml --output corrected.json transcript.json` against a configured +local service. Each invocation captures the files anew. + +## Shared glossary library + +From `examples/campaigns`, `first.yml` and `second.yml` each consume the common +`shared.yml` through the companion `glossary.yml`. Inspect each configuration: + +```sh +audita config validate --config first.yml +audita config validate --config second.yml +``` + +With a local service configured, process either campaign: + +```sh +audita process --config first.yml --output first-output.json transcript.json +audita process --config second.yml --output second-output.json transcript.json +``` + +Edit `prompts.yml` for a local service before processing either campaign. The +first uses separate party and spell corrections; the second selects only party. diff --git a/examples/campaigns/first.yml b/examples/campaigns/first.yml new file mode 100644 index 0000000..f47c4c2 --- /dev/null +++ b/examples/campaigns/first.yml @@ -0,0 +1,8 @@ +version: 3 +llm: {proposal_profile: local} +glossaries: [party, spells] +pipeline: + stages: + - {id: names, module: glossary, glossaries: [party]} + - {id: magic, module: glossary, glossaries: [spells]} + - {module: grammar} diff --git a/examples/campaigns/glossary.yml b/examples/campaigns/glossary.yml new file mode 100644 index 0000000..54a3749 --- /dev/null +++ b/examples/campaigns/glossary.yml @@ -0,0 +1,4 @@ +config: + imports: + - path: shared.yml + required: true diff --git a/examples/campaigns/prompts.yml b/examples/campaigns/prompts.yml new file mode 100644 index 0000000..18ba943 --- /dev/null +++ b/examples/campaigns/prompts.yml @@ -0,0 +1,5 @@ +profiles: + - id: local + endpoint: http://localhost:11434/v1 + model: example-model + api_key_required: false diff --git a/examples/campaigns/second.yml b/examples/campaigns/second.yml new file mode 100644 index 0000000..9f00fcc --- /dev/null +++ b/examples/campaigns/second.yml @@ -0,0 +1,7 @@ +version: 3 +llm: {proposal_profile: local} +glossaries: [party] +pipeline: + stages: + - {id: names, module: glossary} + - {module: grammar} diff --git a/examples/campaigns/shared.yml b/examples/campaigns/shared.yml new file mode 100644 index 0000000..4a0dce1 --- /dev/null +++ b/examples/campaigns/shared.yml @@ -0,0 +1,12 @@ +party: + entries: + - id: lyra + name: Lyra + category: character + description: A fictional adventurer. +spells: + entries: + - id: shield + name: Shield + category: spell + aliases: [Ward] diff --git a/examples/campaigns/transcript.json b/examples/campaigns/transcript.json new file mode 100644 index 0000000..3e7a92f --- /dev/null +++ b/examples/campaigns/transcript.json @@ -0,0 +1,9 @@ +[ + { + "id": 1, + "speaker": "Narrator", + "start": 0, + "end": 2, + "text": "Lyra raises a ward before the journey." + } +] diff --git a/examples/configkit-inheritance.yml b/examples/configkit-inheritance.yml deleted file mode 100644 index 604b7d9..0000000 --- a/examples/configkit-inheritance.yml +++ /dev/null @@ -1,15 +0,0 @@ -version: 2 -profiles_dir: profiles/local -backends: - local-base: - endpoint: http://127.0.0.1:11434/v1 - concurrency: 4 - queue_capacity: 0 - local: - extends: local-base - concurrency: 2 -llm: - proposal_profile: local-proposal - validation_profile: local-validation -pipeline: - modules: [grammar] diff --git a/examples/local/config.yml b/examples/local/config.yml new file mode 100644 index 0000000..b8e8e9f --- /dev/null +++ b/examples/local/config.yml @@ -0,0 +1,8 @@ +version: 3 +llm: + proposal_profile: local +pipeline: + stages: + - module: glossary + - module: grammar +glossaries: [party] diff --git a/examples/local/glossary.yml b/examples/local/glossary.yml new file mode 100644 index 0000000..f4d4d3e --- /dev/null +++ b/examples/local/glossary.yml @@ -0,0 +1,7 @@ +party: + entries: + - id: jesters + name: Jesters + category: faction + description: A traveling group of performers. + aliases: [Jester] diff --git a/examples/local/prompts.yml b/examples/local/prompts.yml new file mode 100644 index 0000000..18ba943 --- /dev/null +++ b/examples/local/prompts.yml @@ -0,0 +1,5 @@ +profiles: + - id: local + endpoint: http://localhost:11434/v1 + model: example-model + api_key_required: false diff --git a/examples/local/transcript.json b/examples/local/transcript.json new file mode 100644 index 0000000..bf27415 --- /dev/null +++ b/examples/local/transcript.json @@ -0,0 +1,9 @@ +[ + { + "id": 1, + "speaker": "Narrator", + "start": 0, + "end": 2, + "text": "The gestures gather beside the stage." + } +] diff --git a/examples/local/validation-candidate.json b/examples/local/validation-candidate.json new file mode 100644 index 0000000..37114b9 --- /dev/null +++ b/examples/local/validation-candidate.json @@ -0,0 +1 @@ +[{"correction_index":1,"original_text":"not open","corrected_text":"open","original_segment_text":"The door is not open.","corrected_segment_text":"The door is open."}] diff --git a/examples/minimal-config.yml b/examples/minimal-config.yml deleted file mode 100644 index 2073c84..0000000 --- a/examples/minimal-config.yml +++ /dev/null @@ -1,13 +0,0 @@ -version: 2 -profiles_dir: profiles/local -backends: - local: - endpoint: http://127.0.0.1:11434/v1 - concurrency: 1 -llm: - proposal_profile: local-proposal - validation_profile: local-validation -pipeline: - modules: [grammar] -output: - schema: bare-segments diff --git a/examples/overlays/testing.yml b/examples/overlays/testing.yml deleted file mode 100644 index ed682f4..0000000 --- a/examples/overlays/testing.yml +++ /dev/null @@ -1,3 +0,0 @@ -llm: - proposal_profile: service-proposal-testing - validation_profile: service-validation-testing diff --git a/examples/production-config.yml b/examples/production-config.yml deleted file mode 100644 index b200c4c..0000000 --- a/examples/production-config.yml +++ /dev/null @@ -1,41 +0,0 @@ -version: 2 -profiles_dir: profiles/service -backends: - service: - endpoint: https://openrouter.ai/api/v1 - credential: AUDITA_SERVICE_KEY - concurrency: 2 - queue_capacity: 1024 -llm: - proposal_profile: service-proposal - validation_profile: service-validation - transient_retries: 2 - repair_attempts: 2 - operation_timeout: 15m -pipeline: - modules: [glossary, homophones, glossary, spoken_word, grammar] -output: - schema: audita-v1 -chunking: - # Requires at least this many normalized segments; omit for automatic chunking. - target_sections: 8 - max_section_tokens: 8192 - min_section_tokens: 2048 -normalization: - max_segment_gap: 4.0 - ellipsis_gap: 3.5 - max_segment_duration: 60.0 - max_segment_tokens: 2048 -thresholds: - glossary: 0.8 - homophones: 0.8 - spoken_word: 0.8 - grammar: 0.8 -context: - description: "General context for domain vocabulary and speaker names." -reporting: - enabled: true - stdout_dir: . -diagnostics: - debug: false - work_dir: /tmp/audita diff --git a/examples/profiles/local/local-proposal.yaml b/examples/profiles/local/local-proposal.yaml deleted file mode 100644 index 3a7b6cc..0000000 --- a/examples/profiles/local/local-proposal.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - local-proposal: - backend: local - model: replace-with-local-model - temperature: 0 - max_tokens: 2048 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/profiles/local/local-validation.yaml b/examples/profiles/local/local-validation.yaml deleted file mode 100644 index a6b5e75..0000000 --- a/examples/profiles/local/local-validation.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - local-validation: - backend: local - model: replace-with-local-model - temperature: 0 - max_tokens: 1024 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/profiles/service/service-proposal-testing.yaml b/examples/profiles/service/service-proposal-testing.yaml deleted file mode 100644 index fdbc56c..0000000 --- a/examples/profiles/service/service-proposal-testing.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - service-proposal-testing: - backend: service - model: replace-with-testing-model - temperature: 0 - max_tokens: 2048 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/profiles/service/service-proposal.yaml b/examples/profiles/service/service-proposal.yaml deleted file mode 100644 index ea4a949..0000000 --- a/examples/profiles/service/service-proposal.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - service-proposal: - backend: service - model: replace-with-provider-model - temperature: 0 - max_tokens: 2048 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/profiles/service/service-validation-testing.yaml b/examples/profiles/service/service-validation-testing.yaml deleted file mode 100644 index bfd69dd..0000000 --- a/examples/profiles/service/service-validation-testing.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - service-validation-testing: - backend: service - model: replace-with-testing-model - temperature: 0 - max_tokens: 1024 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/profiles/service/service-validation.yaml b/examples/profiles/service/service-validation.yaml deleted file mode 100644 index 3e1c741..0000000 --- a/examples/profiles/service/service-validation.yaml +++ /dev/null @@ -1,8 +0,0 @@ -profiles: - service-validation: - backend: service - model: replace-with-provider-model - temperature: 0 - max_tokens: 1024 - top_p: 1 - timeout_seconds: 120 diff --git a/examples/prompt-edit/config.yml b/examples/prompt-edit/config.yml new file mode 100644 index 0000000..b8e8e9f --- /dev/null +++ b/examples/prompt-edit/config.yml @@ -0,0 +1,8 @@ +version: 3 +llm: + proposal_profile: local +pipeline: + stages: + - module: glossary + - module: grammar +glossaries: [party] diff --git a/examples/prompt-edit/glossary.yml b/examples/prompt-edit/glossary.yml new file mode 100644 index 0000000..f4d4d3e --- /dev/null +++ b/examples/prompt-edit/glossary.yml @@ -0,0 +1,7 @@ +party: + entries: + - id: jesters + name: Jesters + category: faction + description: A traveling group of performers. + aliases: [Jester] diff --git a/examples/prompt-edit/prompts.yml b/examples/prompt-edit/prompts.yml new file mode 100644 index 0000000..986f753 --- /dev/null +++ b/examples/prompt-edit/prompts.yml @@ -0,0 +1,23 @@ +profiles: + - id: local + endpoint: http://localhost:11434/v1 + model: example-model + api_key_required: false +collections: + audita: + prompts: + - id: modules.grammar.proposal + messages: + - role: system + content: Correct only clearly supported grammatical errors. + - role: user + content: | + {{ .ProposalContract }} + {{ .HardeningPolicy }} + {{ .TranscriptDescriptionBlock }} + Glossary: {{ .GlossaryJSON }} + Transcript: {{ .SectionJSON }} + output: + format: json + validation_mode: json_schema + schema_path: correction_set.v1.json diff --git a/examples/prompt-edit/transcript.json b/examples/prompt-edit/transcript.json new file mode 100644 index 0000000..bf27415 --- /dev/null +++ b/examples/prompt-edit/transcript.json @@ -0,0 +1,9 @@ +[ + { + "id": 1, + "speaker": "Narrator", + "start": 0, + "end": 2, + "text": "The gestures gather beside the stage." + } +] diff --git a/examples/service/config.yml b/examples/service/config.yml new file mode 100644 index 0000000..9d71f14 --- /dev/null +++ b/examples/service/config.yml @@ -0,0 +1,7 @@ +version: 3 +llm: + proposal_profile: service-proposal + validation_profile: service-review +pipeline: + stages: + - module: grammar diff --git a/examples/service/overlays/testing.yml b/examples/service/overlays/testing.yml new file mode 100644 index 0000000..9d0a423 --- /dev/null +++ b/examples/service/overlays/testing.yml @@ -0,0 +1,3 @@ +llm: + proposal_profile: service-test + validation_profile: service-test diff --git a/examples/service/prompts.yml b/examples/service/prompts.yml new file mode 100644 index 0000000..573b768 --- /dev/null +++ b/examples/service/prompts.yml @@ -0,0 +1,16 @@ +backends: + - id: service + endpoint: https://example.invalid/v1 + api_key_env: AUDITA_EXAMPLE_KEY + concurrency_limit: 2 + queue_capacity: 8 +profiles: + - id: service-proposal + backend: service + model: example-proposal-model + - id: service-review + backend: service + model: example-review-model + - id: service-test + extends: {id: service-proposal} + model: example-testing-model diff --git a/examples/service/transcript.json b/examples/service/transcript.json new file mode 100644 index 0000000..bf27415 --- /dev/null +++ b/examples/service/transcript.json @@ -0,0 +1,9 @@ +[ + { + "id": 1, + "speaker": "Narrator", + "start": 0, + "end": 2, + "text": "The gestures gather beside the stage." + } +] diff --git a/examples/tiny-glossary.yaml b/examples/tiny-glossary.yaml deleted file mode 100644 index c660b40..0000000 --- a/examples/tiny-glossary.yaml +++ /dev/null @@ -1,6 +0,0 @@ -glossary: - - name: Audita - aliases: - - audita - category: product - summary: The Audita transcript correction CLI. diff --git a/internal/cli/examples_integration_test.go b/internal/cli/examples_integration_test.go new file mode 100644 index 0000000..99c2899 --- /dev/null +++ b/internal/cli/examples_integration_test.go @@ -0,0 +1,156 @@ +package cli + +import ( + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "os" + "path/filepath" + "runtime" + "strings" + "sync" + "sync/atomic" + "testing" + + "gitea.maximumdirect.net/eric/audita/v3/internal/core/config" + "gitea.maximumdirect.net/eric/audita/v3/internal/core/schema" + "gitea.maximumdirect.net/eric/audita/v3/internal/framework/llm" + ck "go.maximumdirect.net/configkit" +) + +func TestMaintainedProjectsWithControlledProvider(t *testing.T) { + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("cannot locate examples") + } + root := filepath.Join(filepath.Dir(file), "..", "..", "examples") + t.Setenv("AUDITA_EXAMPLE_KEY", "synthetic-example-key") + for _, tc := range []struct{ dir, main, overlay string }{ + {"local", "config.yml", ""}, {"service", "config.yml", "testing"}, + {"prompt-edit", "config.yml", ""}, {"campaigns", "first.yml", ""}, {"campaigns", "second.yml", ""}, + } { + t.Run(tc.dir+"/"+tc.main, func(t *testing.T) { + main := filepath.Join(root, tc.dir, tc.main) + loaded, err := config.Load(ck.Input{Source: ck.File(main)}, tc.overlay) + if err != nil { + t.Fatal(err) + } + loaded.Config.Reporting.Enabled = false + var calls atomic.Int32 + var mu sync.Mutex + var models, prompts []string + transport := controlledTransport(func(r *http.Request) (*http.Response, error) { + calls.Add(1) + var request struct { + Model string `json:"model"` + Messages []struct { + Content string `json:"content"` + } `json:"messages"` + } + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + return nil, err + } + var content strings.Builder + for _, message := range request.Messages { + content.WriteString(message.Content) + } + mu.Lock() + models = append(models, request.Model) + prompts = append(prompts, content.String()) + mu.Unlock() + return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(`{"model":"fixture","choices":[{"message":{"role":"assistant","content":"{\"corrections\":[]}"}}]}`)), Request: r}, nil + }) + engine, err := llm.NewRuntime(context.Background(), loaded, &http.Client{Transport: transport}) + if err != nil { + t.Fatal(err) + } + if tc.dir == "prompt-edit" { + prompt, err := engine.Engine.InspectPrompt(context.Background(), "audita/modules.grammar.proposal") + if err != nil || !strings.Contains(prompt.Messages[0].Content, "clearly supported grammatical errors") { + t.Fatalf("operator prompt: %+v %v", prompt, err) + } + } + raw, err := os.ReadFile(filepath.Join(root, tc.dir, "transcript.json")) + if err != nil { + t.Fatal(err) + } + source, err := schema.ParseSourceTranscriptJSON(raw) + if err != nil { + t.Fatal(err) + } + inputs := &processInputs{Configuration: loaded, Runtime: engine, Source: source} + prepared, err := prepareProcess(context.Background(), processInput{Transcript: filepath.Join(root, tc.dir, "transcript.json")}, inputs, &bytes.Buffer{}) + if err != nil { + t.Fatal(err) + } + defer prepared.Prepared.Output.Close() + if _, err := executePipeline(context.Background(), inputs, prepared, sourceToCanonicalTranscript(source), func(error) {}); err != nil { + t.Fatal(err) + } + if int(calls.Load()) != len(loaded.Config.Pipeline.Stages) { + t.Fatalf("provider calls=%d", calls.Load()) + } + if tc.dir == "service" && (len(models) != 1 || models[0] != "example-testing-model") { + t.Fatalf("overlay models=%v", models) + } + if tc.dir == "campaigns" && tc.main == "first.yml" { + if len(prompts) != 3 || !strings.Contains(prompts[0], `"name": "Lyra"`) || strings.Contains(prompts[0], `"name": "Shield"`) || !strings.Contains(prompts[1], `"name": "Shield"`) || strings.Contains(prompts[1], `"name": "Lyra"`) { + t.Fatalf("campaign glossary selections: %v", prompts) + } + } + if tc.dir == "campaigns" && tc.main == "second.yml" { + if len(prompts) != 2 || !strings.Contains(prompts[0], `"name": "Lyra"`) || strings.Contains(prompts[0], `"name": "Shield"`) { + t.Fatalf("second campaign glossary selection: %v", prompts) + } + } + }) + } +} + +func TestExamplePromptEditIsCapturedOnNextLoad(t *testing.T) { + _, file, _, ok := runtime.Caller(0) + if !ok { + t.Fatal("cannot locate examples") + } + source := filepath.Join(filepath.Dir(file), "..", "..", "examples", "prompt-edit") + dir := t.TempDir() + for _, name := range []string{"config.yml", "glossary.yml", "prompts.yml"} { + raw, err := os.ReadFile(filepath.Join(source, name)) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, name), raw, 0600); err != nil { + t.Fatal(err) + } + } + inspect := func() string { + loaded, err := config.Load(ck.Input{Source: ck.File(filepath.Join(dir, "config.yml"))}, "") + if err != nil { + t.Fatal(err) + } + engine, err := llm.NewRuntime(context.Background(), loaded, nil) + if err != nil { + t.Fatal(err) + } + prompt, err := engine.Engine.InspectPrompt(context.Background(), "audita/modules.grammar.proposal") + if err != nil { + t.Fatal(err) + } + return prompt.Messages[0].Content + } + before := inspect() + filePath := filepath.Join(dir, "prompts.yml") + raw, err := os.ReadFile(filePath) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filePath, []byte(strings.Replace(string(raw), "clearly supported grammatical errors", "clearly supported punctuation errors", 1)), 0600); err != nil { + t.Fatal(err) + } + after := inspect() + if !strings.Contains(before, "grammatical errors") || !strings.Contains(after, "punctuation errors") { + t.Fatalf("before=%q after=%q", before, after) + } +}