Files
ansible-workspace-dnd/docs/roadmap/features/0022-coordinated-suite-upgrade.md

301 lines
16 KiB
Markdown
Raw Permalink Normal View History

# Feature 0022: Integrate the Reviewed Application Upgrades
Status: done
## Summary
Bring the four reviewed standalone upgrades into the D&D pipeline role,
collection, and deployment consumer as one compatible configuration set.
This completes the agreed
[application upgrade sequence](../application-upgrade-sequence.md).
## Problem or Opportunity
The standalone roles now support the new releases, but the suite intentionally
retains its previous pins and configuration. Audita still receives v1 settings;
Narratio and Notarius testing selection still uses retired profile terminology.
Updating executable versions alone would break the deployed configuration.
## Scope
- Adopt the reviewed application/role versions together.
- Provide Audita v2 backend configuration and suite-owned PromptKit profiles.
- Migrate Narratio credential forwarding and Narratio/Notarius testing overlays.
- Preserve D&D processing, artifact composition, storage/publishing, shared
filesystem, and operator-access behavior.
- Update the collection snapshot, public dependency manifests, consumer pins,
operator guidance, and focused offline integration coverage.
## Non-goals
- Deploying to the operational controller or `pipeline-dnd` host, running real
transcripts, or calling inference, WhisperX, or object-storage services.
- New models, prompts, artifact families, extraction lanes, or processing policy
beyond migration choices expressly approved here.
- Automatic manifest conversion, cache deletion, forced stage reruns, broad
directory cleanup, compatibility aliases, or runtime version/integrity probes.
- Changing generic standalone-role ownership or adding production validation.
- A newly tagged collection release or replacement of published 0.1.0 assets.
This feature updates the public development compatibility set; formal release
packaging remains a separate decision.
## Target End State
### Coordinated Versions and Distribution
| Application | Suite version | Reviewed standalone role revision |
| --- | --- | --- |
| Scriptorium | v0.15.0 | `b174c8c` |
| Audita | v2.0.1 | `c3dd59f` |
| Notarius | v0.8.1 | `6aef48b` |
| Narratio | v3.0.0 | `3c0700b` |
| Seriatim | v1.5.0, unchanged | existing v0.1.0 role pin |
Dependency manifests use full immutable revisions resolved from these reviewed
commits. The pipeline role remains authoritative; the collection imports its
completed revision through the established public Gitea subtree workflow, never
by editing `roles/pipeline` directly. The collection continues bundling the
pipeline role and declaring standalone role dependencies separately.
Relevant reviewed commits must be published to their existing public origins
before downstream manifests/subtree imports depend on them. Publication of the
required Git commits is part of the eventual implementation workflow, not this
roadmap-writing pass. Do not rewrite tags or publish unrelated changes.
The consumer keeps its minimal playbook and host-vars organization. Its role
and collection pins select the same compatibility set. No controller install
or remote playbook execution is an acceptance gate.
### Audita Backend and Profile Ownership
The pipeline role supplies `audita_backends`, profile selection, and schema-2
settings to the standalone Audita role. The pipeline owns the missing profile
assets under `/usr/local/etc/audita/profiles/`, using its runtime owner/group and
configuration modes. This directory contains PromptKit LLM profiles, not
ConfigKit overlays; it must not be renamed to `overlays`.
Approved suite interface:
- `dnd_pipeline_audita_backends`: native backend map, defaulting to one backend
named `dnd`, endpoint `https://inference.ai.rakestrawhome.com/v1`, credential
reference `AUDITA_LLM_API_KEY`, and shared concurrency `4`. Omit queue capacity
to inherit the native default.
- `dnd_pipeline_audita_profiles`: a native profile mapping keyed by file/profile
ID, with complete `dnd-proposal` and `dnd-validation` entries. Each selects
backend `dnd`, model `google/gemma-4-31b-it`, and per-attempt
`timeout_seconds: 900`. Other model controls inherit native behavior.
- `dnd_pipeline_audita_llm_proposal_profile` and
`dnd_pipeline_audita_llm_validation_profile` select those respective IDs.
Render each supplied profile as one safely serialized native mapping with its
matching ID. Document profile-map keys as filename-safe IDs. Do not introduce
per-provider templates, profile merging, or discovery of external assets.
Callers can compose native maps in inventory and replace the supplied maps.
As with existing managed secrets, removing a map entry does not imply pruning
arbitrary files; only selected profiles participate in normal execution.
Preserve the ordered module sequence, `audita-v1` output, chunking values,
thresholds, context description, and normalization limits. Convert normalization
duration defaults from `3s`, `3s`, and `240s` to numeric `3.0`, `3.0`, and
`240.0` seconds. Preserve meaningful setting names where native semantics are
unchanged; update argument types accordingly.
Remove retired per-role endpoint/model/credential/concurrency/retry variables
in favor of the native backend/profile interface. Document their replacements;
do not infer a conversion from arbitrary old inventory. Remove diagnostics
retention; leave debug disabled through native defaults. Retain the existing
diagnostics work-directory choice where still supported. Reporting follows
Audita's native configuration; Narratio consumes its ordinary output contract.
Production and testing continue using the same Audita endpoint and model.
No separate Audita testing overlay is needed for an identical configuration.
### Credential Lists and Overlay Selection
Replace the suite's singular Narratio Notarius credential variable with
`dnd_pipeline_narratio_notarius_credentials`, default `[OPENROUTER_API_KEY]`.
Add `dnd_pipeline_narratio_audita_credentials`, default `[AUDITA_LLM_API_KEY]`.
Pass both through the reviewed Narratio role's native list variables. These
lists contain names, never values, and can be replaced with `[]`. Operators
overriding Audita backends must explicitly list all referenced credentials;
the suite does not inspect configurations to discover names.
Preserve `dnd_pipeline_secrets`, existing secret names, Vault references,
protected secret directory/files, and ordinary world-readable nonsecret
configuration. No credential rotation, secret relocation, or forwarding logic
is required in Ansible.
Deploy the suite-owned testing files at:
- `/usr/local/etc/narratio/overlays/testing.yml`
- `/usr/local/etc/notarius/overlays/testing.yml`
Narratio testing sets `notarius.config_overlay: testing`; production omits the
selector. Notarius remains enabled in both modes. Its production pipeline
continues selecting `gpt-terra-latest`, testing selects `gpt-luna-latest`, and
the existing caller-overridable pipeline map/ID behavior remains intact.
Scriptorium testing still selects `gemini-flash-latest`; production continues
using the prompt assets' selections. Retain separate artifact/family fragments.
Use Narratio `--overlay testing` and Notarius `--config-overlay testing` in
tests and operator examples. Deploy the new files declaratively; leave ignored
legacy `profiles/testing.yml` files alone and document that they are obsolete.
Do not delete `profiles/` directories or conflate them with PromptKit profiles.
### Runtime Paths and Existing Suite Behavior
Keep Narratio's explicit downstream binaries/config paths, output handoffs,
Notarius `/var/cache/notarius` roots and resume behavior, shared writable
directories/ACLs, service user, operator enrollment, and S3/publishing choices.
Audita profiles are configuration assets, not shared output directories.
No new runtime storage architecture is required.
### Retry and Timeout Policy
Retain the previously selected per-attempt Audita profile timeouts of 900
seconds while inheriting the new applications' retry,
repair, and whole-operation defaults. Expose nullable suite pass-throughs for
`audita_llm_transient_retries`, `audita_llm_repair_attempts`,
`audita_llm_operation_timeout`, `notarius_promptkit_transient_retries`, and
`scriptorium_transient_retries`, prefixed with `dnd_pipeline_` in the public
suite interface. Null means omit; explicit zero is preserved where valid.
Do not mechanically translate old `max_retries: 3` into both transport retries
and structural repairs. Preserve explicitly authored Notarius module retries;
they are a separate behavior. Keep Narratio stage timeouts unchanged. Explain
the nested budgets and the changed semantics, not a guarantee of identical
provider cost or timing.
## Repository Ownership
- `ansible-role-dnd-pipeline`: suite defaults, argument specs, dependency wiring,
Audita profile assets, testing overlays, documentation, and integration tests.
- `ansible-collection-dnd`: imported pipeline snapshot, dependency manifest,
collection tests, and compatibility guidance.
- `rakestraw-ansible`: matching dependency pins and deployment guidance; adjust
host variables only where required. Preserve Vault content and unrelated
inventory. Keep the hosts-and-roles playbook minimal.
- Standalone role repositories: reviewed commits are reused/published, not
redesigned. Application and campaign/prompt repositories are read-only
contracts/assets unless a concrete incompatibility is raised for discussion.
- Workspace: roadmap, later staged plan, decisions, progress, and review.
## Constraints and Settled Decisions
Follow the [Ansible](../../ansible.md), [testing](../../testing.md), and
[planning](../../planning.md) policies, [MVP roadmap](../mvp.md), and
[upgrade sequence](../application-upgrade-sequence.md). New D&D policy remains
in the pipeline role; optional generic application defaults remain null.
## Simplest Viable Implementation
Update dependency variables/pins and existing templates, add an ordinary
Audita profile directory task and one looped template deployment, and change
testing destinations to `overlays/`. Reuse existing declarative convergence,
secret delivery, and collection assembly. No production validation, state
fingerprints, migration framework, or generated credential inference is needed.
## Acceptance Criteria
- Pipeline and collection installations produce a compatible set of the four
upgraded applications while Seriatim remains unchanged.
- Native offline inspection/validation accepts Audita settings and selected
profiles, Notarius base/testing configurations, Scriptorium configuration,
and Narratio base/testing composition with synthetic credential inputs.
Direct Audita validation receives credential environment variables explicitly:
it does not bootstrap Narratio's secrets directory on its own.
- Focused cases prove caller-supplied backend/profile configuration and
credential lists, testing selection, and existing important composition and
path handoffs. Assert supplied values, not fixed copies of today's defaults.
- Immediate repeated convergence is clean. Existing meaningful shared-user,
cache, artifact-composition, and installed-collection behavior remains covered
without expanding into upstream module/runtime regression matrices.
- Collection build/install and consumer syntax checks use the established
public dependency manifests. No sibling application/role checkout is needed
by an external consumer. Required referenced commits exist on public Gitea.
- Consumer inventory retains comments/Vault references for unknown secrets;
no actual keys or operational deployments are required for completion.
- Guidance covers breaking inventory fields, new CLI selectors, obsolete
testing-file locations, and upstream manifest migration. Back up and migrate
local provenance without deleting progress or editing immutable snapshots;
document normal named-selector invalidation rather than forcing reruns.
## Open Questions
None. The user approved deferring to application defaults wherever possible,
with nullable inventory overrides, and confirmed retaining the existing
900-second Audita per-attempt policy. Do not freeze native retry, repair, or
operation-deadline defaults in suite variables. Operators should note that an
attempt may consume the entire logical-operation budget; these are separate
controls, not additive timeout guarantees.
## Dependencies and Follow-up Work
Features 0018–0021 provide the reviewed standalone revisions. Implementation,
independent review, and remediation are complete. Git history preserves the
completed staged implementation plan. Actual deployment,
session migration, live processing, and a formal tagged release are separate
operator actions, not completion gates for this feature.
## Implementation Evidence and Review Handoff
On 2026-09-29, the four reviewed standalone role revisions were published to
their existing public Gitea `origin/main` branches and verified after fetch:
Scriptorium `b174c8c2990bb79f8149d8f288c5e06ccb42bd73`, Audita
`c3dd59f5ac9b942306ffe540ae9e71d493998137`, Notarius
`6aef48b7e12e3651971444b8c224d96ff1879e6c`, and Narratio
`3c0700b5323293d44e3b1e67a50074d82d7a3bff`.
The pipeline role at public revision
`ea8b09e190419586ede6df3815d1ea999a0f4391` passed `bash -n`, its full
`tests/integration` harness twice (configuration migration and representative
override coverage), and `git diff --check`. The collection imported that exact
public revision by subtree; public collection revision
`3d77eb382bcb4c1742015d3c32e23630bb8c73bf` passed `bash -n`, its full
artifact-based `tests/integration`, and `git diff --check`. Both harnesses used
synthetic credentials and offline native application validation, with clean
repeats. The consumer's local commit
`f694dad9426ee21138b175e44a3855d42be99d4b` passed public dependency
installation into disposable paths, real-inventory `--list-hosts` limited to
`pipeline-dnd`, `--syntax-check`, and `git diff --check`. Its commit was not
pushed. Seriatim and native collection pins remain unchanged, and manifests
across the three repositories agree.
The local integration harnesses converged only the development installation;
there was no operational deployment or live inference, transcription, or object
storage call. Vault values and target access were not needed or verified.
After the override harness passed, a formatting-only split of its fixture JSON
line was checked with `bash -n`, JSON parsing, and `git diff --check` without
rerunning the long integration harness. Independent review subsequently found
the three issues resolved below. No tagged release was created.
## Remediation Follow-up
The three independent-review findings are resolved; see
[remediation evidence](../remediation.md). Public pipeline `37b1cf6` and
collection `b3bc045` remove default-identity assertions and correct migration
and credential guidance. Local consumer `15f0f4e` selects the corrected public
snapshot. Shell/whitespace checks, pipeline and installed-collection syntax and
lint, collection build/install, and real-inventory consumer target/syntax checks
passed. Full host-mutating convergence was not rerun for these documentation
and test-only changes. Application versions/defaults and runtime behavior are
unchanged.
## Closeout
Closed on 2026-09-29 after independent review and resolution of all three
findings. This completes the agreed application upgrade sequence. The active
implementation plan is retired; no additional implementation is required for
this feature. Consumer commits remain local. Operational upgrade, session
migration, live production/testing checks, and any new tagged release require
separate operator action and were not performed as part of closeout.
## Upstream References
- [Audita v2 migration](https://gitea.maximumdirect.net/eric/audita/src/tag/v2.0.1/docs/migrations/v2.md)
- [Audita configuration](https://gitea.maximumdirect.net/eric/audita/src/tag/v2.0.1/docs/config.md)
- [Narratio v3 release](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/releases/v3.0.0.md)
- [Narratio operations and migration](https://gitea.maximumdirect.net/eric/narratio/src/tag/v3.0.0/docs/operations.md)
- [Notarius v0.8.0 migration](https://gitea.maximumdirect.net/eric/notarius/src/tag/v0.8.0/docs/releases/v0.8.0.md)
- [Scriptorium v0.14.0 migration](https://gitea.maximumdirect.net/eric/scriptorium/src/tag/v0.14.0/docs/releases/v0.14.0.md)