Plan the ConfigKit upgrade and workspace development
This commit is contained in:
84
docs/adr/0001-classify-session-content.md
Normal file
84
docs/adr/0001-classify-session-content.md
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
# 0001. Classify session content by purpose and lifecycle
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposed
|
||||||
|
|
||||||
|
## Date
|
||||||
|
|
||||||
|
2026-10-01
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Narratio coordinates applications that produce transcripts, artifacts,
|
||||||
|
diagnostics, and working files. The current session layout mixes some of these
|
||||||
|
roles, including complete Notarius bundles under `artifacts/`. Operators use
|
||||||
|
transcripts and artifacts to evaluate results during development and need
|
||||||
|
predictable locations for diagnostics.
|
||||||
|
|
||||||
|
A shared classification model will guide the workspace design and subsequent
|
||||||
|
adapter changes. The [operator guide](../operations.md#local-state-layout)
|
||||||
|
describes the implemented layout.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Classify content by its primary purpose and lifecycle:
|
||||||
|
|
||||||
|
| Category | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| Inputs | Material consumed by an application, including audio, configuration, and reference documents. |
|
||||||
|
| Outputs | Completed substantive products, including intermediate artifacts exposed for further use. |
|
||||||
|
| Logs | Records of execution, including process logs, execution reports, warnings, and diagnostic summaries. |
|
||||||
|
| Cache | Reusable information whose absence permits ordinary recomputation or retrieval. |
|
||||||
|
| State | Application condition, progress, recovery records, and unfinished work. |
|
||||||
|
|
||||||
|
Classification is relative to the consuming application. An output of one stage
|
||||||
|
can serve as an input to another while retaining its output location. Completed
|
||||||
|
outputs reused to skip work remain Outputs. Narratio's manifest is State and
|
||||||
|
records whether those outputs are reusable.
|
||||||
|
|
||||||
|
Classify checkpoints according to their recovery behavior. A checkpoint needed
|
||||||
|
to continue a particular interrupted operation is State. A saved work result
|
||||||
|
that can be reused or recomputed is Cache. Recomputation cost informs retention
|
||||||
|
policy. File-specific classification remains necessary for stores serving both
|
||||||
|
purposes.
|
||||||
|
|
||||||
|
Treat Debug as a capture and retention mode that can include detailed Logs,
|
||||||
|
snapshots of State, and other diagnostic evidence. Retained debug snapshots have
|
||||||
|
a diagnostic role; live recovery records retain their State role.
|
||||||
|
|
||||||
|
Apply these directory principles to Narratio:
|
||||||
|
|
||||||
|
- Store completed artifacts, including intermediate artifacts like
|
||||||
|
`session_bounds.json`, directly in a flat `artifacts/` directory.
|
||||||
|
- Store combined transcripts directly in `transcripts/`, with a subdirectory
|
||||||
|
for the initial per-speaker transcripts.
|
||||||
|
- Store prepared external audio in `audio/` and other prepared external inputs
|
||||||
|
in `inputs/`.
|
||||||
|
- Organize Logs under `logs/` by stage. Execution reports belong to this category.
|
||||||
|
- Organize Cache under a cache root by the application that owns its format and
|
||||||
|
reuse behavior.
|
||||||
|
- Treat scratch files and unfinished outputs as State until completion permits
|
||||||
|
promotion to Outputs.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
- Group outputs by producer. This exposes subprocess packaging to operators;
|
||||||
|
flat artifact and transcript locations better support comparison and review.
|
||||||
|
- Group Logs by application. Stage names match the operator's workflow and
|
||||||
|
collect diagnostics from stages that invoke multiple applications.
|
||||||
|
- Classify all checkpoints as State. Recovery behavior provides a more useful
|
||||||
|
distinction for eviction and reuse decisions.
|
||||||
|
- Treat all Debug material as Logs. A separate debug location can accommodate
|
||||||
|
retained state snapshots and larger diagnostic bundles when needed.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Each managed file family needs an identified role, owner, scope, and retention
|
||||||
|
rule. Subprocess bundles require classification before their contents are
|
||||||
|
placed in Narratio's output and diagnostic directories.
|
||||||
|
|
||||||
|
Physical roots, retention rules, the mechanism for presenting latest logs,
|
||||||
|
debug placement, and ambiguous checkpoint classifications remain open. The file
|
||||||
|
inventory and subsequent roadmap will resolve these choices and define the
|
||||||
|
migration. Current versus historical views are separate from content categories.
|
||||||
76
docs/adr/0002-assign-subprocess-content-ownership.md
Normal file
76
docs/adr/0002-assign-subprocess-content-ownership.md
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
# 0002. Assign subprocess content ownership
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposed
|
||||||
|
|
||||||
|
## Date
|
||||||
|
|
||||||
|
2026-10-03
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
[ADR 0001](0001-classify-session-content.md) defines content categories and
|
||||||
|
operator-facing directory principles. Narratio also needs an ownership boundary
|
||||||
|
for the applications it invokes. That boundary determines who selects locations,
|
||||||
|
manages retained files, and controls cleanup.
|
||||||
|
|
||||||
|
The current subprocess workflows can execute with fresh working state. Notarius
|
||||||
|
can reuse persistent chunk-plan and checkpoint caches. Narratio maintains its own
|
||||||
|
session progress and recovery records.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Assign responsibility by content category:
|
||||||
|
|
||||||
|
| Category | Responsibility |
|
||||||
|
| --- | --- |
|
||||||
|
| Inputs | Narratio supplies paths to prepared inputs, transcripts, and artifacts. |
|
||||||
|
| Outputs | Narratio selects destinations and promotes completed products into the output locations defined by ADR 0001. |
|
||||||
|
| Logs | Narratio selects stage-based destinations and captures subprocess output streams. |
|
||||||
|
| Debug | Narratio selects the destination when capture is enabled; the subprocess defines the diagnostic contents. |
|
||||||
|
| Subprocess State | Each application manages its working directory, temporary files, and execution lifecycle through its own configuration and defaults. |
|
||||||
|
| Subprocess Cache | Each application manages its configured persistent cache, including its format, identity, and reuse behavior. |
|
||||||
|
|
||||||
|
Narratio retains ownership of its own State, including the session manifest,
|
||||||
|
recovery bookkeeping, and staging files it manages.
|
||||||
|
|
||||||
|
Recommend a dedicated cache root for each cache-aware application, configurable
|
||||||
|
through that application's supported controls. For system-wide deployments,
|
||||||
|
`/var/cache/<application>` is a suggested location; `/var/cache/notarius` is an
|
||||||
|
example for Notarius. Each application defines its cache configuration, defaults,
|
||||||
|
and internal layout. Operators select locations and retention settings through
|
||||||
|
those controls. Narratio's session cleanup leaves subprocess caches under their
|
||||||
|
separate lifecycle policies.
|
||||||
|
|
||||||
|
Subprocesses retain ownership of their file formats and bundle structures.
|
||||||
|
Narratio can receive a complete result bundle at a selected staging destination
|
||||||
|
and promote the selected products into its canonical output directories.
|
||||||
|
Diagnostic snapshots of subprocess State follow the Debug destination selected
|
||||||
|
by Narratio.
|
||||||
|
|
||||||
|
Narratio's `runs/` directory can hold staging outputs and retained execution
|
||||||
|
evidence. Subprocess working directories follow the application's configuration
|
||||||
|
and may reside elsewhere. Narratio cleanup manages Narratio-owned material;
|
||||||
|
subprocess State and Cache follow their separate lifecycle policies.
|
||||||
|
|
||||||
|
## Alternatives Considered
|
||||||
|
|
||||||
|
- Place all subprocess working files and caches inside each Narratio session.
|
||||||
|
This makes sessions easier to collect as a unit, but couples application
|
||||||
|
internals and cache retention to Narratio's session lifecycle.
|
||||||
|
- Allow subprocesses to choose output, log, and debug destinations independently.
|
||||||
|
This simplifies invocation but scatters the material operators inspect.
|
||||||
|
- Give Narratio control of downstream bundle formats. This simplifies some
|
||||||
|
destination mappings but increases integration coupling. Selecting and
|
||||||
|
promoting products preserves application ownership of those formats.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Adapters need supported controls for Narratio-selected output, log, and debug
|
||||||
|
locations. Upstream changes may be needed where those destinations are coupled
|
||||||
|
to application working directories or to one another.
|
||||||
|
|
||||||
|
Workspace layout and file classification can proceed independently of subprocess
|
||||||
|
working-directory conventions. Exact debug placement, retained diagnostic views,
|
||||||
|
and migration steps remain for the workspace design and roadmap.
|
||||||
175
docs/roadmap/audita.md
Normal file
175
docs/roadmap/audita.md
Normal file
@@ -0,0 +1,175 @@
|
|||||||
|
# Adopt Audita v3 and independent diagnostic destinations
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Adopt Audita v3.0.0 for transcript polishing and use its independent output
|
||||||
|
destinations to implement the Audita portion of the
|
||||||
|
[session workspace roadmap](workspace.md). Operators should find the completed
|
||||||
|
transcript, ordinary reports, and optional debug evidence in their respective
|
||||||
|
Narratio locations.
|
||||||
|
|
||||||
|
This document defines the Audita adoption scope and target. The workspace
|
||||||
|
roadmap owns shared layout, latest-attempt views, and retention behavior. Its
|
||||||
|
referenced ADRs remain Proposed. Implementation sequencing belongs in a
|
||||||
|
subsequent implementation plan.
|
||||||
|
|
||||||
|
## Reviewed Upstream Release
|
||||||
|
|
||||||
|
The reviewed Audita v3.0.0 tag identifies commit
|
||||||
|
`3b4933256eff84327b812e7ae12babf743bc04a6`. Its
|
||||||
|
[release notes](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/releases/v3.0.0.md),
|
||||||
|
[CLI reference](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/cli.md),
|
||||||
|
[configuration reference](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/config.md),
|
||||||
|
and [operations guide](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/operations.md)
|
||||||
|
own the downstream contracts.
|
||||||
|
|
||||||
|
Audita now honors an explicit `--report-dir` alongside file transcript output.
|
||||||
|
Its existing `--work-dir` selects the parent of retained debug captures, with
|
||||||
|
capture enabled separately through configuration or `--debug`. These controls
|
||||||
|
satisfy Narratio's requested destination separation. Further upstream changes
|
||||||
|
are unnecessary for this scope.
|
||||||
|
|
||||||
|
## Current Baseline
|
||||||
|
|
||||||
|
Narratio's [Audita integration](../integrations/audita.md) and
|
||||||
|
[polish stage](../internal/stage-polish.md) describe the implemented behavior:
|
||||||
|
|
||||||
|
- The adapter invokes `process` with a transcript, glossary, and staged transcript
|
||||||
|
destination. It supports independent downstream configuration, overlay,
|
||||||
|
profile, module, description, and output-schema selectors.
|
||||||
|
- The adapter supplies no reporting, debug, or work-directory flags. Audita
|
||||||
|
controls debug capture and placement through its own configuration.
|
||||||
|
- Ordinary reports are discovered beside the staged transcript. The polish
|
||||||
|
stage validates returned report paths against those adjacent filenames.
|
||||||
|
- Only the validated polished transcript becomes a canonical stage output.
|
||||||
|
Reports, captured streams, and the generated invocation record are diagnostics.
|
||||||
|
- Selected credential references use Narratio's shared exact-name forwarding and
|
||||||
|
captured-stream redaction boundary.
|
||||||
|
|
||||||
|
The existing invocation remains usable with Audita v3 after deployment-side
|
||||||
|
profile migration. Narratio changes are needed to request and consume the
|
||||||
|
new destination separation.
|
||||||
|
|
||||||
|
## Targeted End State
|
||||||
|
|
||||||
|
### Products and diagnostics
|
||||||
|
|
||||||
|
Narratio supplies separate destinations for each actual polish attempt:
|
||||||
|
|
||||||
|
| Content | Destination and control |
|
||||||
|
| --- | --- |
|
||||||
|
| Corrected transcript | Run-local staging via `--output`; validated and promoted to `transcripts/polished.json` on success. |
|
||||||
|
| Ordinary reports | `runs/<run_id>/polish/logs/` via `--report-dir`. |
|
||||||
|
| Captured stdout, stderr, and Narratio invocation record | The same run-local polish log directory. |
|
||||||
|
| Optional debug evidence | `runs/<run_id>/polish/debug/` via `--work-dir`; Audita may create its own `run-*` subdirectory beneath it. |
|
||||||
|
|
||||||
|
The workspace roadmap's `logs/polish` and `debug/polish` links expose the latest
|
||||||
|
attempt. Narratio supplies actual run destinations to Audita and records those
|
||||||
|
paths in manifest diagnostics. Shared link updates and cleanup protection are
|
||||||
|
implemented by the workspace work rather than separately inside the adapter.
|
||||||
|
|
||||||
|
### Capture selection
|
||||||
|
|
||||||
|
Add an optional `pipeline.audita.debug` boolean, defaulting to false. A true
|
||||||
|
value passes `--debug`. False or omission supplies no capture override and
|
||||||
|
preserves Audita configuration and overlay behavior; it does not disable debug
|
||||||
|
enabled downstream.
|
||||||
|
|
||||||
|
Supply `--work-dir` for each executing attempt so downstream-enabled capture
|
||||||
|
also lands in the Narratio-selected debug directory. Selecting that destination
|
||||||
|
alone does not enable capture. Ordinary production operation retains Audita's
|
||||||
|
default reporting and debug settings. Narratio adds no reporting enable/disable
|
||||||
|
flag or debug-disable flag.
|
||||||
|
|
||||||
|
Debug selection affects diagnostics for an executing stage. Changing it alone
|
||||||
|
does not invalidate a completed transcript. Operators use the existing force
|
||||||
|
workflow when fresh polish or debug evidence is needed.
|
||||||
|
|
||||||
|
### Ownership and compatibility
|
||||||
|
|
||||||
|
Use a clean adoption of Audita v3's CLI contract, without version-dependent
|
||||||
|
adapter branches or legacy profile conversion. Audita continues to own its
|
||||||
|
configuration, profiles, prompts, runtime State, and diagnostic contents.
|
||||||
|
Narratio owns supplied destinations and transcript promotion.
|
||||||
|
|
||||||
|
Keep existing selectors and shared credentials such as `OPENROUTER_API_KEY` and
|
||||||
|
`RAKESTRAWHOME_API_KEY` under their configured exact names. Audita's examples
|
||||||
|
do not require Narratio to adopt application-specific credential aliases.
|
||||||
|
Operators continue to list the credential references required by all configured
|
||||||
|
Audita backends.
|
||||||
|
|
||||||
|
Audita is a separately installed executable. Its adoption of ConfigKit v0.5.0
|
||||||
|
and PromptKit v0.12.0 does not require changing Narratio's own library
|
||||||
|
dependencies as part of this feature.
|
||||||
|
|
||||||
|
## Required Changes
|
||||||
|
|
||||||
|
### Deployment configuration migration
|
||||||
|
|
||||||
|
Document the manual migration of Audita-owned PromptKit files:
|
||||||
|
|
||||||
|
- Wrap profile definitions under keyed `profiles:` mappings, remove inner `id`
|
||||||
|
fields, and replace `base_profile` with `extends`. Preserve selected profile
|
||||||
|
IDs when appropriate.
|
||||||
|
- Review explicit zeros, omitted values, and nulls against the new generation
|
||||||
|
control semantics. Validate the final configuration root, including unused
|
||||||
|
definitions and backend references.
|
||||||
|
- Rename any backend literally named `extends` and update its references.
|
||||||
|
- Validate production configuration and each used testing overlay with Audita's
|
||||||
|
offline `config validate` command. Audita application configuration continues
|
||||||
|
to use `version: 2`.
|
||||||
|
|
||||||
|
Keep the detailed downstream format owned by the pinned Audita documentation.
|
||||||
|
Narratio does not inspect, rewrite, or automatically migrate these files.
|
||||||
|
Deployment can migrate Audita before the workspace changes land.
|
||||||
|
|
||||||
|
### Adapter and polish stage
|
||||||
|
|
||||||
|
Carry explicit report and debug destinations through the Audita request and
|
||||||
|
construct the supported flags inside the adapter. Wire the debug selector
|
||||||
|
through configuration and runner composition.
|
||||||
|
|
||||||
|
Discover the existing three report filenames in the requested report directory.
|
||||||
|
Update polish's expected-path validation to the same location. Update invocation
|
||||||
|
records and safe result metadata to describe the selected destinations and
|
||||||
|
capture request. Preserve available failure evidence and the original subprocess
|
||||||
|
error.
|
||||||
|
|
||||||
|
Keep existing transcript validation, report read bounds, timeouts, credential
|
||||||
|
forwarding, and captured-stream redaction. Reports remain optional diagnostics;
|
||||||
|
early failures or downstream-disabled reporting may produce none. Exit-zero
|
||||||
|
runs with recoverable model errors remain successful. A nonzero exit, including
|
||||||
|
a late report-write failure, never authorizes transcript promotion. Narratio
|
||||||
|
does not add report-schema parsing or report-status success gates.
|
||||||
|
|
||||||
|
### Workspace integration and lifecycle
|
||||||
|
|
||||||
|
Use the shared workspace path owner for run-local log/debug destinations and
|
||||||
|
the agreed latest-attempt behavior. Preserve previous views on skipped stages;
|
||||||
|
actual attempts refresh both views, including on failure. An attempt without
|
||||||
|
debug capture exposes an empty current debug directory.
|
||||||
|
|
||||||
|
Keep full Audita debug bundles local in this feature. Preserve existing declared
|
||||||
|
diagnostic publication and apply the workspace roadmap's retention protection
|
||||||
|
to latest link targets. This feature inherits that roadmap's clean session
|
||||||
|
cutover and unchanged treatment of older completed outputs after invalidation
|
||||||
|
or failed replacement.
|
||||||
|
|
||||||
|
### Documentation and verification
|
||||||
|
|
||||||
|
When implemented, update the configuration reference and examples, Audita
|
||||||
|
integration contract, polish internals, and operator guide. Reference Audita v3's
|
||||||
|
profile migration and explain the shared Audita/Notarius overlay development
|
||||||
|
workflow. Downstream configuration edits behind unchanged selectors still
|
||||||
|
require explicitly forcing the affected stage to replace reused outputs.
|
||||||
|
|
||||||
|
Revise the workspace roadmap's upstream-request description when this adoption
|
||||||
|
is incorporated; Audita v3.0.0 supplies the previously missing capability.
|
||||||
|
|
||||||
|
Use focused offline tests for separate transcript/report/debug destinations,
|
||||||
|
enable-only debug selection, capture enabled by downstream configuration,
|
||||||
|
available failure reports, and preservation of nonzero-exit behavior. Verify
|
||||||
|
that reports remain diagnostics and only a successful validated transcript is
|
||||||
|
promoted. Shared latest-view and cleanup tests belong to the workspace change.
|
||||||
|
Default Narratio tests use subprocess doubles and temporary filesystems and
|
||||||
|
remain independent of sibling checkouts, real credentials, and paid providers.
|
||||||
202
docs/roadmap/config.md
Normal file
202
docs/roadmap/config.md
Normal file
@@ -0,0 +1,202 @@
|
|||||||
|
# Adopt ConfigKit v0.5.0
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Upgrade Narratio from ConfigKit v0.4.2 to v0.5.0 and integrate its native
|
||||||
|
sibling inheritance with Narratio's configuration validation and inspection.
|
||||||
|
Keep the upgrade focused on the existing loading model and the adjustments
|
||||||
|
needed to consume the new release correctly.
|
||||||
|
|
||||||
|
This adoption precedes [Audita adoption](audita.md) and the
|
||||||
|
[workspace reorganization](workspace.md). This document defines the target
|
||||||
|
and required changes; [the implementation plan](implementation.md) defines
|
||||||
|
the steps for reaching that target.
|
||||||
|
|
||||||
|
## Reviewed Upstream Release
|
||||||
|
|
||||||
|
ConfigKit v0.5.0 identifies commit
|
||||||
|
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. The
|
||||||
|
[release notes](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/releases/v0.5.0.md),
|
||||||
|
[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/config.md),
|
||||||
|
and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/consumers/README.md)
|
||||||
|
own the upstream behavior.
|
||||||
|
|
||||||
|
The release introduces native `extends` relationships between sibling mappings,
|
||||||
|
layered loading with ordered underlays, caller-owned filesystem and mapping
|
||||||
|
sources, and passive source-resource contexts. Inheritance preserves explicit
|
||||||
|
presence, provenance, and secret classification.
|
||||||
|
|
||||||
|
The existing loading API remains available. ConfigKit's minimum Go line, YAML
|
||||||
|
dependency, and LGPL-3.0-or-later license are unchanged. Narratio can retain its
|
||||||
|
current Go toolchain and ordinary loading calls.
|
||||||
|
|
||||||
|
## Current Baseline
|
||||||
|
|
||||||
|
The [configuration reference](../config.md) owns Narratio's current public
|
||||||
|
semantics; [configuration internals](../internal/configuration.md) describe
|
||||||
|
loading and inspection.
|
||||||
|
|
||||||
|
- Pipeline loading uses ConfigKit with explicit additive imports and one
|
||||||
|
selected overlay, typed defaults, recorded resolution, and domain validation.
|
||||||
|
- Campaign, session, and canonical-party inputs use separate single-source
|
||||||
|
loads. Secret-directory bootstrap remains an explicit invocation operation.
|
||||||
|
- Fields with root-relative pipeline path semantics retain that base when their
|
||||||
|
declarations move into imports. For example, enabled Notarius configuration
|
||||||
|
and working-directory paths resolve from the root pipeline directory.
|
||||||
|
- Narratio adds a declaration-ownership check: each artifact or artifact-family
|
||||||
|
key may be declared by only one base file. Overlay modifications are allowed.
|
||||||
|
- Source inspection and artifact catalogs consume retained ConfigKit metadata.
|
||||||
|
Effective show, diff, and digest projections describe the resolved domain
|
||||||
|
configuration rather than source layout.
|
||||||
|
|
||||||
|
The review selected v0.5.0 through a temporary module file and ran Narratio's
|
||||||
|
existing test suite successfully. An additional probe exposed an integration
|
||||||
|
gap: a child artifact inheriting from a parent in another imported file is
|
||||||
|
rejected as declared by multiple base files. The same definitions in one file
|
||||||
|
load successfully. The check currently counts inherited history contributions
|
||||||
|
as if they were direct declarations.
|
||||||
|
|
||||||
|
## Targeted End State
|
||||||
|
|
||||||
|
### Native inheritance
|
||||||
|
|
||||||
|
Use ConfigKit's built-in sibling inheritance wherever its input contract and
|
||||||
|
Narratio's domain schema permit it. ConfigKit resolves `extends` after source
|
||||||
|
composition and overlay precedence, before typed decoding and application
|
||||||
|
resolution. Narratio receives the resulting typed configuration.
|
||||||
|
|
||||||
|
Inheritance can reduce repetition between artifacts and artifact families,
|
||||||
|
including definitions in separate imported files. Distinct child keys retain
|
||||||
|
their identities. Each resulting entry undergoes ordinary Narratio validation,
|
||||||
|
family expansion, dependency resolution, and execution selection.
|
||||||
|
|
||||||
|
Parent entries remain ordinary domain entries. They must satisfy applicable
|
||||||
|
validation and participate in the effective configuration. This adoption does
|
||||||
|
not introduce abstract artifacts or a separate template registry. Operators
|
||||||
|
must override inherited fields where needed to keep the resulting configuration
|
||||||
|
valid, such as output destinations.
|
||||||
|
|
||||||
|
Inherited false, zero, empty, and null input retains ConfigKit's explicit
|
||||||
|
presence semantics. Narratio's absence-based defaults must not refill such
|
||||||
|
values. Normal domain validation still determines whether a resulting value
|
||||||
|
is acceptable.
|
||||||
|
|
||||||
|
### Declaration ownership and provenance
|
||||||
|
|
||||||
|
Distinguish direct base declarations from inherited contributions. A child
|
||||||
|
declared in one file may inherit from a separately declared parent in another
|
||||||
|
file without becoming a duplicate declaration.
|
||||||
|
|
||||||
|
Retain the existing rejection of one artifact or family key directly declared
|
||||||
|
by multiple base files, including disjoint contributions and declarations
|
||||||
|
later removed by an overlay. Inheritance is not permission to merge multiple
|
||||||
|
direct declarations of the same key.
|
||||||
|
|
||||||
|
Keep defining-file ownership distinct from value-contributor provenance.
|
||||||
|
Existing source and artifact inspection must describe inherited contributors
|
||||||
|
accurately without presenting them as conflicting declarations. ConfigKit's
|
||||||
|
retained metadata remains the source of that evidence.
|
||||||
|
|
||||||
|
For generated family members, the family's defining source identifies the
|
||||||
|
child family declaration; inherited parent files remain contributor evidence.
|
||||||
|
Keep existing logical artifact IDs and inspection interfaces.
|
||||||
|
|
||||||
|
Effective configuration, diffs, and digests continue to reflect resolved
|
||||||
|
domain values. Equivalent inherited and explicitly authored configurations
|
||||||
|
should have equivalent effective projections, regardless of their provenance.
|
||||||
|
Stage reuse continues to use existing semantic evidence.
|
||||||
|
|
||||||
|
### Loading and ownership boundaries
|
||||||
|
|
||||||
|
Retain ordinary ConfigKit loading, centralized typed defaults, imports, selected
|
||||||
|
overlays, domain resolution, credential bootstrap, and exact-name credential
|
||||||
|
forwarding. Preserve current path bases and read-only inspection workflows.
|
||||||
|
|
||||||
|
Layered loading, embedded configuration sources, programmatic mapping sources,
|
||||||
|
and source-relative resource rebasing are available upstream but are outside
|
||||||
|
this focused adoption. Narratio continues to own its domain rules; ConfigKit
|
||||||
|
owns structural inheritance, presence, provenance, and secret resolution.
|
||||||
|
|
||||||
|
## Required Changes
|
||||||
|
|
||||||
|
### Dependency and attribution
|
||||||
|
|
||||||
|
Update `go.mod` and `go.sum` to ConfigKit v0.5.0. Update its entry in
|
||||||
|
[attribution](../oss/attribution.md) to the selected version and the reviewed
|
||||||
|
commit's source and notice links. Existing applicable license texts can remain.
|
||||||
|
|
||||||
|
### Configuration integration
|
||||||
|
|
||||||
|
Adjust the existing duplicate-definition check to identify direct declarations
|
||||||
|
using public ConfigKit history and origin metadata. Preserve genuine duplicate
|
||||||
|
detection while excluding structural transfers from declaration counting.
|
||||||
|
Use the completed load's retained evidence rather than reparsing files or
|
||||||
|
performing a second load that could resolve superseded credential references.
|
||||||
|
|
||||||
|
Review source ownership and artifact-definition projections against the new
|
||||||
|
inheritance metadata. Update only where needed to report contributors correctly.
|
||||||
|
Keep effective rendering and digest construction grounded in the existing safe
|
||||||
|
typed projection, with credential identities rather than resolved values.
|
||||||
|
|
||||||
|
Use ConfigKit's native operation directly. No application-owned inheritance
|
||||||
|
engine, collection registration, alternative keyword, or compatibility loader
|
||||||
|
is needed.
|
||||||
|
|
||||||
|
### Compatibility and operator guidance
|
||||||
|
|
||||||
|
Document that `extends` is now reserved structural input throughout ConfigKit
|
||||||
|
loads, including dynamic mappings such as artifact IDs and prompt variables.
|
||||||
|
Operators using that key as ordinary data must rename it. Inheritance is active
|
||||||
|
without an opt-in switch; imports and overlay-file selection remain separately
|
||||||
|
controlled as before.
|
||||||
|
|
||||||
|
Explain native sibling lookup, inherited explicit presence, and the requirement
|
||||||
|
that parent and child entries satisfy Narratio's domain rules. Link to the
|
||||||
|
pinned upstream contract for detailed inheritance semantics. Clarify that a
|
||||||
|
selected overlay may contribute sibling inheritance directives; this does not
|
||||||
|
add inheritance or stacking between overlay files.
|
||||||
|
|
||||||
|
Update the configuration reference, internal configuration guide, and a small
|
||||||
|
maintained example when implementation lands. Preserve each field's current
|
||||||
|
path-resolution rule when its value is inherited. Record the reserved-key
|
||||||
|
compatibility impact in the eventual release note; automatic configuration
|
||||||
|
conversion is outside this feature.
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
Add focused Narratio coverage for the confirmed cross-file artifact inheritance
|
||||||
|
gap and the corresponding artifact-family behavior. Retain coverage for real
|
||||||
|
duplicate declarations, including those hidden by overlay replacement or
|
||||||
|
removal. Verify inherited explicit values against Narratio's defaulting rules,
|
||||||
|
credential safety, contributor reporting, and equivalent effective projections.
|
||||||
|
|
||||||
|
Concentrate tests on Narratio's integration behavior. ConfigKit's own tests own
|
||||||
|
its generic inheritance algorithm and source lifecycle. Narratio's default
|
||||||
|
tests remain offline, use temporary configuration bundles and controlled secret
|
||||||
|
values, and do not depend on the sibling ConfigKit checkout. Run the existing
|
||||||
|
suite against the selected released module before completing the upgrade.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
Narratio selects the released ConfigKit v0.5.0 module without a local checkout
|
||||||
|
replacement. Cross-file artifact and family inheritance works through the
|
||||||
|
ordinary loader, while genuinely duplicated base declarations remain invalid.
|
||||||
|
Inspection distinguishes declaration ownership from inherited contributors;
|
||||||
|
effective projections and credential handling retain their existing semantics.
|
||||||
|
|
||||||
|
Maintained guidance and examples explain native inheritance and the reserved
|
||||||
|
`extends` key. Attribution identifies the selected upstream revision. Focused
|
||||||
|
regression coverage and the existing suite pass without implementing Audita or
|
||||||
|
workspace changes as part of this upgrade.
|
||||||
|
|
||||||
|
## Relationship To Subsequent Work
|
||||||
|
|
||||||
|
ConfigKit adoption provides a verified configuration baseline for the later
|
||||||
|
Audita and workspace features. It makes no changes to subprocess destinations,
|
||||||
|
session layout, or cleanup behavior.
|
||||||
|
|
||||||
|
Audita adoption then migrates downstream profiles and adds adapter support for
|
||||||
|
explicit report/debug destinations. Audita owns its separate ConfigKit and
|
||||||
|
PromptKit dependencies. Shared latest-attempt links, final workspace paths, and
|
||||||
|
retention protection remain part of the workspace reorganization, so those
|
||||||
|
facilities are implemented once at their shared owner.
|
||||||
319
docs/roadmap/implementation.md
Normal file
319
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,319 @@
|
|||||||
|
# ConfigKit v0.5.0 implementation plan
|
||||||
|
|
||||||
|
## Scope and authority
|
||||||
|
|
||||||
|
Implement [the ConfigKit roadmap](config.md). It owns the target, compatibility
|
||||||
|
choices, and rationale. This plan covers Narratio's upgrade from v0.4.2 to
|
||||||
|
v0.5.0 and integration with native sibling inheritance. Audita adoption and
|
||||||
|
workspace reorganization remain subsequent, separately planned work.
|
||||||
|
|
||||||
|
The reviewed upstream revision is
|
||||||
|
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`, tagged `v0.5.0`.
|
||||||
|
Use the released module, not a local replacement. The pinned
|
||||||
|
[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/config.md)
|
||||||
|
and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/consumers/README.md)
|
||||||
|
own upstream semantics. Read their sibling-inheritance and metadata sections
|
||||||
|
when working on the relevant stages; source may also be inspected in the
|
||||||
|
selected Go module cache. The sibling checkout is optional reference material,
|
||||||
|
not a build or test dependency.
|
||||||
|
|
||||||
|
## Working instructions
|
||||||
|
|
||||||
|
The audience is a Luna coding agent. Complete the numbered stages in order.
|
||||||
|
Each stage is a bounded unit with its own verification and completion criteria.
|
||||||
|
Inspect current code before editing; the paths below identify owners rather
|
||||||
|
than requiring changes to every listed file.
|
||||||
|
|
||||||
|
- Start with `docs/development.md`, `docs/policy/agents.md`, and `config.md` in
|
||||||
|
this directory. Follow the development policy-reading guide. Architecture,
|
||||||
|
testing, documentation, and OSS policies apply to this work; release policy
|
||||||
|
applies if discussing compatibility or eventual release preparation.
|
||||||
|
- Use Go 1.26. Prefer the repository's graph tools for code discovery and
|
||||||
|
inspect focused tests at each package boundary.
|
||||||
|
- Keep ordinary `Load`, typed defaults, explicit imports, one selected overlay,
|
||||||
|
domain validation, and current path-resolution rules. Native inheritance is
|
||||||
|
available wherever ConfigKit and Narratio's schema permit it, with no new
|
||||||
|
collection allowlist, opt-in flag, or Narratio-owned inheritance engine.
|
||||||
|
- Distinguish original declarations from inherited contributors. Preserve
|
||||||
|
genuine duplicate-base rejection and truthful provenance. Parent entries
|
||||||
|
remain ordinary entries subject to existing domain rules.
|
||||||
|
- Keep invocation-owned secrets bootstrap and shared exact-name credential
|
||||||
|
forwarding unchanged. Do not introduce a second base load, file reparse, or
|
||||||
|
historical-secret lookup to recover declaration information.
|
||||||
|
- Tests use temporary files, controlled synthetic secrets, and real internal
|
||||||
|
collaborators. No downstream executable, sibling checkout, cloud service,
|
||||||
|
or paid provider is needed. Protect integration behavior rather than
|
||||||
|
duplicating ConfigKit's generic algorithm tests or freezing incidental output.
|
||||||
|
- Commit each completed nonempty unit with a short plain-English message after
|
||||||
|
suitable validation. Report blockers and incomplete work accurately. If a
|
||||||
|
review-only task needs no source fix, retain useful behavioral coverage and
|
||||||
|
report that result rather than making a cosmetic rewrite.
|
||||||
|
|
||||||
|
Do not adopt layered loading, new source types, source-relative rebasing,
|
||||||
|
abstract artifacts, automatic YAML conversion, new execution selection rules,
|
||||||
|
new output collision policy, or changes to subprocess credentials and workspace
|
||||||
|
paths in this feature. Existing domain validation and reuse rules still apply;
|
||||||
|
do not add prohibitions, hardening, or guarantees merely because they can be
|
||||||
|
tested. This plan does not authorize a release, push, tag, ADR acceptance, or
|
||||||
|
retirement of pending roadmaps.
|
||||||
|
|
||||||
|
## Stage 1: Select the released dependency and update attribution
|
||||||
|
|
||||||
|
Purpose: establish the actual dependency baseline before integration work.
|
||||||
|
|
||||||
|
Read `go.mod`, `go.sum`, `docs/oss/attribution.md`, the OSS policy, and the
|
||||||
|
selected release's notes, `COPYRIGHT`, and `LICENSE`.
|
||||||
|
|
||||||
|
1. Select `gitea.maximumdirect.net/eric/ConfigKit@v0.5.0` with the normal Go
|
||||||
|
module workflow. Keep Go 1.26 and avoid unrelated dependency upgrades.
|
||||||
|
2. Update ConfigKit's attribution entry to v0.5.0 and use the full reviewed
|
||||||
|
commit for its Source, COPYRIGHT, and LICENSE links. Preserve upstream
|
||||||
|
copyright wording, SPDX expression, and applicable local license links.
|
||||||
|
Its license and YAML dependency are unchanged; no new license copy is needed.
|
||||||
|
3. Inspect module changes for unintended churn or a local `replace`. Existing
|
||||||
|
configuration loaders should compile without adopting optional APIs.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config ./internal/app ./internal/doccheck`
|
||||||
|
and `go mod tidy -diff`. Confirm
|
||||||
|
`go list -m gitea.maximumdirect.net/eric/ConfigKit` selects v0.5.0 without a
|
||||||
|
replacement.
|
||||||
|
|
||||||
|
Complete when the released dependency and notices agree and current focused
|
||||||
|
tests pass. The cross-file inheritance defect is addressed in the next stage;
|
||||||
|
this stage does not claim that new inheritance behavior is fully integrated.
|
||||||
|
|
||||||
|
## Stage 2: Correct direct declaration accounting
|
||||||
|
|
||||||
|
Purpose: allow distinct artifact and family keys to inherit across imported
|
||||||
|
files while preserving existing duplicate-declaration rejection.
|
||||||
|
|
||||||
|
Read `internal/config/pipeline_configkit.go`, particularly
|
||||||
|
`pipelineMetadataFromConfigKit` and `rejectDuplicateBaseDefinitions`, and
|
||||||
|
`internal/config/pipeline_configkit_test.go`. Inspect the public ConfigKit
|
||||||
|
`Metadata.History`, `HistoricalPaths`, `Origin`, and `ChangeKind` contracts.
|
||||||
|
|
||||||
|
1. Add package-boundary regression fixtures with a valid parent in one imported
|
||||||
|
file and a child using `extends` in another. Cover ordinary artifacts and
|
||||||
|
artifact families. Give parent and child distinct valid output paths or
|
||||||
|
family patterns, and provide canonical-party context for expanded families.
|
||||||
|
Both keys remain ordinary domain entries; child IDs must stay their own.
|
||||||
|
2. Adjust declaration accounting using retained public metadata. In v0.5.0,
|
||||||
|
original base-input events have `LayerBase` and no origin operation;
|
||||||
|
projected transfers carry `ChangeInheritance`. Count original base
|
||||||
|
declarations, not these transfers or subsequent decoding/resolution events.
|
||||||
|
Consumed structural `extends` paths are not artifact declarations.
|
||||||
|
3. Keep the historical-path check, including original declarations erased by
|
||||||
|
a selected overlay. Do not switch to surviving effective keys only or count
|
||||||
|
all displaced origins as fresh declarations. Reuse a small package-local
|
||||||
|
helper only if it makes the next stage clearer.
|
||||||
|
4. Retain `TestPipelineConfigKitRejectsDuplicateBaseDeclarations`, including
|
||||||
|
disjoint fields supplied to the same key, container removal, and removal of
|
||||||
|
Scriptorium by an overlay. An overlay may still modify an existing entry.
|
||||||
|
Add only any missing regression case needed to exercise the new filter.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config`. The positive cross-file regression
|
||||||
|
should expose the baseline defect before the fix. Assert loaded values and
|
||||||
|
identities, or the duplicate category and relevant sources, without matching an
|
||||||
|
entire diagnostic or ConfigKit's private event ordering.
|
||||||
|
|
||||||
|
Complete when cross-file inheritance loads and expands correctly, and actual
|
||||||
|
duplicates still fail even when the overlay hides them. Do not use the direct
|
||||||
|
declaration filter to remove contributor evidence from inspection.
|
||||||
|
|
||||||
|
## Stage 3: Preserve contributor provenance and family ownership
|
||||||
|
|
||||||
|
Purpose: keep inspection useful when one entry contains values from several
|
||||||
|
files, without confusing its defining file with those contributors.
|
||||||
|
|
||||||
|
Read `internal/config/artifact_catalog.go`,
|
||||||
|
`internal/config/artifact_families.go`, `internal/config/effective_pipeline.go`,
|
||||||
|
and metadata-to-ownership projection in `pipeline_configkit.go`. Inspect the
|
||||||
|
focused catalog, family, and effective-source tests.
|
||||||
|
|
||||||
|
1. Exercise the inherited configurations from Stage 2 through
|
||||||
|
`EffectiveArtifactCatalog` and `EffectivePipelineSources`. Check inherited
|
||||||
|
values against their actual parent contributors and child overrides against
|
||||||
|
the child/overlay contributors. Keep historical definition evidence distinct
|
||||||
|
from effective field ownership.
|
||||||
|
2. Preserve `DefinitionSources` aggregation, including inherited contributors.
|
||||||
|
Do not relabel all inherited values as authored in the child's file.
|
||||||
|
3. Correct `artifactFamilySource` where its first inherited field currently
|
||||||
|
selects the parent's file. For an explicitly declared child family, identify
|
||||||
|
its original declaration using retained metadata: use its base defining file
|
||||||
|
when present, or its overlay defining file for an overlay-only family.
|
||||||
|
Overlay edits to a base family do not relocate its base definition.
|
||||||
|
4. Retain the existing fallback for entries without a direct declaration or
|
||||||
|
programmatically constructed configuration. Do not restrict valid enclosing
|
||||||
|
inheritance merely to force every entry into a single-file ownership model.
|
||||||
|
5. Keep logical `SourceID`, artifact keys, family/character/party provenance,
|
||||||
|
existing source roles, and inspection interfaces. Update projections only
|
||||||
|
where inherited evidence is otherwise lost or incorrectly attributed.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config`. Cover an explicitly declared child
|
||||||
|
family inheriting from another file and a selected-overlay contribution. Check
|
||||||
|
meaningful records and source identities, not a complete formatted snapshot.
|
||||||
|
|
||||||
|
Complete when contributor reporting remains truthful and generated members
|
||||||
|
identify their child family definition rather than an arbitrary inherited field.
|
||||||
|
|
||||||
|
## Stage 4: Verify inherited presence, domain resolution, and path bases
|
||||||
|
|
||||||
|
Purpose: prevent Narratio's application defaults or normalization from undoing
|
||||||
|
the library's inherited explicit input.
|
||||||
|
|
||||||
|
Read `pipelineConfigDefaults`, `resolvePipelineConfig`, and
|
||||||
|
`internal/config/load.go`, including `finalizeLoadedPipeline` and
|
||||||
|
`resolveNotariusPaths`. Inspect the current presence/defaulting tests and
|
||||||
|
the relevant domain validators.
|
||||||
|
|
||||||
|
1. Add representative inheritance cases for explicit false, zero, empty,
|
||||||
|
and null input versus genuinely absent input. Include a pointer boolean
|
||||||
|
and an entry-level absence-based default where Narratio consumes metadata.
|
||||||
|
For example, inherited `render_debug: false` can exercise artifact presence;
|
||||||
|
inherited empty Scriptorium timeout can exercise an existing invalid value
|
||||||
|
that must not silently become its default.
|
||||||
|
2. Preserve the existing domain outcome for each chosen field. Valid explicit
|
||||||
|
values retain their value; explicit values already invalid must still fail
|
||||||
|
ordinary validation rather than be refilled. An empty mapping merges under
|
||||||
|
ConfigKit's rules and must not be described as universal container removal.
|
||||||
|
3. Exercise overlay composition before inheritance: modify a parent's field
|
||||||
|
in the selected overlay and confirm the child's inherited result uses that
|
||||||
|
effective value. A child override still wins over its parent.
|
||||||
|
4. Verify a known root-relative field, such as enabled Notarius `config_path`
|
||||||
|
or `working_directory`, when inherited from an imported declaration.
|
||||||
|
Resolve it exactly as the existing field contract does. Do not use passive
|
||||||
|
SourceContext to rebase it to the parent's or child's source directory, or
|
||||||
|
newly normalize other path fields merely to make their behavior uniform.
|
||||||
|
5. Fix any demonstrated Narratio integration defect at its existing defaulting
|
||||||
|
or domain-resolution owner. Use ConfigKit presence and recorded operations;
|
||||||
|
do not recreate structural inheritance or weaken ordinary validation.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config`. Keep coverage representative; generic
|
||||||
|
missing-parent, cycle, resource-context, and expansion-limit matrices belong to
|
||||||
|
ConfigKit. Parents must remain valid Narratio entries, not special templates.
|
||||||
|
|
||||||
|
Complete when inherited explicit input and absent defaults have distinct,
|
||||||
|
correct domain outcomes and existing path bases are unchanged.
|
||||||
|
|
||||||
|
## Stage 5: Verify safe effective projections and inherited credentials
|
||||||
|
|
||||||
|
Purpose: ensure inheritance changes neither semantic comparison nor the
|
||||||
|
existing secret boundary.
|
||||||
|
|
||||||
|
Read `internal/config/effective_pipeline.go`, `effective_digest.go`,
|
||||||
|
`effective_projection.go`, `scriptorium_credentials_test.go`, and the effective
|
||||||
|
projection tests in `load_validate_test.go`. For diff composition, read
|
||||||
|
`internal/app/config_diff.go`. Reuse
|
||||||
|
`TestPipelineOverlayDoesNotRequireUnusedBaseCredentials` and
|
||||||
|
`TestCredentialValuesNeverEnterEffectiveProjectionOrDigest` coverage.
|
||||||
|
|
||||||
|
1. Load two configurations that produce the same effective domain values:
|
||||||
|
one uses inheritance, the other writes the expanded values explicitly.
|
||||||
|
Include ordinary artifacts and representative resolved family members.
|
||||||
|
Verify equal safe effective mappings and digests and no effective diff.
|
||||||
|
Do not compare private provenance as part of semantic equality.
|
||||||
|
2. Add a representative typed credential inheritance case using shared fields
|
||||||
|
of valid Narratio sibling mappings and controlled synthetic environment
|
||||||
|
variables. Verify resolved credentials retain their exact configured names;
|
||||||
|
serialization and inspection expose those identities without values.
|
||||||
|
3. Verify overlay removal/replacement of a credential before inheritance does
|
||||||
|
not resolve the superseded reference. Reuse the existing selected-overlay
|
||||||
|
regression rather than creating a second loader or bootstrap algorithm.
|
||||||
|
4. Check that changing only synthetic secret values leaves semantic projections
|
||||||
|
and digest unchanged. Fix only demonstrated projection defects; preserve
|
||||||
|
the existing safe typed rendering boundary and semantic reuse evidence.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config ./internal/app`. Existing shared adapter
|
||||||
|
credential-forwarding tests remain the owner of subprocess forwarding; do not
|
||||||
|
rewrite adapters or duplicate those tests simply because input can be inherited.
|
||||||
|
|
||||||
|
Complete when resolved values, not source layout or secret bytes, determine
|
||||||
|
effective comparison, and inherited credentials obey the existing safe boundary.
|
||||||
|
|
||||||
|
## Stage 6: Document inheritance and maintain a small example
|
||||||
|
|
||||||
|
Purpose: make the adopted behavior understandable and usable without expanding
|
||||||
|
Narratio's configuration surface.
|
||||||
|
|
||||||
|
Read `docs/config.md`, `docs/internal/configuration.md`, `examples/README.md`,
|
||||||
|
the production/testing bundle, and `TestExamplesLoadAndValidate` in
|
||||||
|
`internal/config/load_validate_test.go`. Follow the documentation policy.
|
||||||
|
|
||||||
|
1. Add native sibling inheritance guidance to the configuration reference.
|
||||||
|
Explain exact sibling lookup, overlay-before-inheritance resolution,
|
||||||
|
explicit presence, ordinary parent validation, distinct child identity,
|
||||||
|
and cross-file declaration/contributor distinctions. Link to the pinned
|
||||||
|
upstream contract for detailed syntax and structural rules.
|
||||||
|
2. Explain the reserved `extends` key throughout ConfigKit inputs, including
|
||||||
|
dynamic artifact IDs and prompt-variable mappings. Operators must rename
|
||||||
|
existing ordinary-data keys with that name; there is no opt-in or automatic
|
||||||
|
conversion. Campaign, session, and party inputs also use ConfigKit.
|
||||||
|
3. Clarify the existing statement that overlays cannot inherit: overlay files
|
||||||
|
cannot inherit or stack, but the selected overlay may supply native sibling
|
||||||
|
directives within its values. Keep imports and overlay selection unchanged.
|
||||||
|
4. Update configuration internals for inheritance before typed decoding,
|
||||||
|
original-declaration counting, contributor projection, and the safe
|
||||||
|
effective boundary. Describe implemented mechanisms, linking to the public
|
||||||
|
reference for their operator-facing meaning.
|
||||||
|
5. In `examples/production-testing/overlays/testing.yml`, add
|
||||||
|
`extends: session_recap` to the existing `testing_notes` artifact. Keep its
|
||||||
|
disabled status and distinct output override. It inherits the imported
|
||||||
|
parent's prompt/input and the parent's testing profile after overlay
|
||||||
|
application. The parent remains an ordinary artifact.
|
||||||
|
6. Update the example guide and extend existing example validation to check
|
||||||
|
those inherited fields. Preserve the bundle's current identities and
|
||||||
|
artifact count; no new example root or template convention is necessary.
|
||||||
|
|
||||||
|
Verify with `go test ./internal/config -run '^TestExamplesLoadAndValidate$'`
|
||||||
|
and `go test ./internal/doccheck`. Review examples for synthetic-only content
|
||||||
|
and validate local links. Keep unimplemented Audita/workspace behavior in its
|
||||||
|
roadmaps. Do not create a release note under an invented Narratio version;
|
||||||
|
reserved-key compatibility impact remains an explicit release-preparation item.
|
||||||
|
|
||||||
|
Complete when canonical documentation and a tested copyable example describe
|
||||||
|
the implemented adoption, without promoting new restrictions to policy.
|
||||||
|
|
||||||
|
## Stage 7: Verify the assembled inspection workflow and close the feature
|
||||||
|
|
||||||
|
Purpose: establish that real command composition consumes the integrated
|
||||||
|
configuration correctly and finish the upgrade as a reviewable baseline.
|
||||||
|
|
||||||
|
Read `internal/app/config_commands_test.go`, `config_artifacts_test.go`,
|
||||||
|
and existing read-only inspection workflow coverage. Reuse the preceding
|
||||||
|
package coverage rather than repeating its entire matrix at the CLI boundary.
|
||||||
|
|
||||||
|
1. Add or extend a representative command-level temporary imported bundle with
|
||||||
|
inheritance. Exercise validation, effective show, source reporting, artifact
|
||||||
|
catalog, and diff through existing command entry points. Use a campaign/party
|
||||||
|
fixture for family inspection, and paired overlays with equivalent inherited
|
||||||
|
and explicitly expanded values when testing command-level diff equality.
|
||||||
|
2. Check semantic output: resolved child values, expected contributor/defining
|
||||||
|
sources, logical artifact identities, and no secret values. Keep inspection
|
||||||
|
read-only with respect to session/runtime state, as existing tests establish.
|
||||||
|
Do not invoke downstream applications or add CLI modes or columns.
|
||||||
|
3. Review the combined diff against `config.md` and this plan. Confirm that
|
||||||
|
ConfigKit v0.5.0 is selected, attribution links use the reviewed revision,
|
||||||
|
no optional loading feature was adopted, and no Audita/workspace change or
|
||||||
|
compatibility shim slipped into scope.
|
||||||
|
4. Run `go test ./...`, `go test -race ./...`, `go vet ./...`,
|
||||||
|
`go build ./...`, `go mod tidy -diff`, and `git diff --check` against the
|
||||||
|
final source. These reuse existing repository checks, not new release gates.
|
||||||
|
The full suite includes example and documentation-link validation; focused
|
||||||
|
checks need not be repeated solely for this stage if unchanged.
|
||||||
|
5. Commit the completed unit and report the dependency revision, checks,
|
||||||
|
compatibility impact, and any unfinished work. Leave the pending Audita and
|
||||||
|
workspace plans and Proposed ADRs intact. Retire completed planning files
|
||||||
|
only during separately requested cleanup/release preparation, repairing links.
|
||||||
|
|
||||||
|
Complete when the feature roadmap's completion criteria are evidenced by the
|
||||||
|
released dependency, integrated behavior, maintained documentation/examples,
|
||||||
|
and passing final checks. The next implementation scope is Audita; its concrete
|
||||||
|
plan is prepared separately. Release version selection and publication are
|
||||||
|
outside this implementation pass.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
None required for this adoption. Existing decisions select native inheritance,
|
||||||
|
current loading/path/credential semantics, and ConfigKit before Audita before
|
||||||
|
workspace work. Eventual release preparation must account for the reserved-key
|
||||||
|
compatibility impact under the release policy; no version is selected here.
|
||||||
280
docs/roadmap/workspace.md
Normal file
280
docs/roadmap/workspace.md
Normal file
@@ -0,0 +1,280 @@
|
|||||||
|
# Session workspace organization
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Give operators predictable locations for session inputs, completed outputs,
|
||||||
|
logs, and debug evidence. Apply the taxonomy in
|
||||||
|
[ADR 0001](../adr/0001-classify-session-content.md) and the ownership boundary in
|
||||||
|
[ADR 0002](../adr/0002-assign-subprocess-content-ownership.md).
|
||||||
|
|
||||||
|
This roadmap defines the target behavior and required changes. Implementation
|
||||||
|
sequencing belongs in a subsequent implementation plan. The ADRs remain
|
||||||
|
Proposed pending their acceptance.
|
||||||
|
|
||||||
|
## Confirmed Requirements
|
||||||
|
|
||||||
|
- Completed artifacts occupy a flat `artifacts/` directory. Intermediate
|
||||||
|
products such as `session_bounds.json` belong here.
|
||||||
|
- Combined transcripts occupy `transcripts/` directly. Initial per-speaker
|
||||||
|
transcripts occupy a subdirectory beneath it.
|
||||||
|
- Prepared external audio occupies `audio/`; other prepared external inputs
|
||||||
|
occupy `inputs/`. Outputs consumed by later stages retain their output paths.
|
||||||
|
- Logs and Debug use matching latest-attempt views: `logs/<stage>` and
|
||||||
|
`debug/<stage>` are symlinks to the producing run's directories. Both views
|
||||||
|
advance when a stage starts an actual attempt, including attempts that fail.
|
||||||
|
A skipped stage leaves its previous views intact.
|
||||||
|
- Execution reports and invocation records are Logs. Historical Logs and Debug
|
||||||
|
remain under their producing runs, subject to cleanup.
|
||||||
|
- Narratio can selectively request supported subprocess debug capture through
|
||||||
|
configuration. Omission preserves application configuration and defaults;
|
||||||
|
Narratio needs no explicit downstream debug-disable override.
|
||||||
|
- Prepared session audio is the single retained Narratio-managed audio copy.
|
||||||
|
Remove Narratio's separate persistent audio cache and spool duplicates.
|
||||||
|
- Configurations that produce colliding artifact names or destinations fail
|
||||||
|
validation before execution. Validation covers the effective pipeline,
|
||||||
|
including expanded character families and built-in artifact destinations.
|
||||||
|
- Narratio selects input, output, log, and debug paths. Subprocess applications
|
||||||
|
own their working State and Cache through their supported configuration.
|
||||||
|
- A dedicated cache root such as `/var/cache/notarius` is a deployment
|
||||||
|
recommendation. The application owns its cache defaults, internal layout,
|
||||||
|
identity, and reuse semantics. Narratio session cleanup respects that boundary.
|
||||||
|
- Narratio retains its own manifest, recovery records, and managed staging State.
|
||||||
|
- This is a clean cutover for newly prepared sessions. Existing-session migration
|
||||||
|
and backward compatibility shims are outside the feature.
|
||||||
|
- Treatment of older completed files after invalidation or failed replacement
|
||||||
|
remains unchanged, including their availability through manifest-backed
|
||||||
|
resolution. Directory reorganization does not authorize new stale-file
|
||||||
|
deletion, automatic fallback, or rollback behavior.
|
||||||
|
|
||||||
|
## Current Baseline
|
||||||
|
|
||||||
|
The [operator guide](../operations.md#local-state-layout) owns the current
|
||||||
|
physical layout. Relevant differences from the target are:
|
||||||
|
|
||||||
|
- Extraction moves the complete Notarius bundle to
|
||||||
|
`artifacts/notarius/<run_id>/`. Resume evidence and artifact resolution depend
|
||||||
|
on that bundle and its index.
|
||||||
|
- Merge and normalize promote Seriatim execution reports into `artifacts/`.
|
||||||
|
- Analysis output paths and character-family patterns can introduce nested
|
||||||
|
artifact directories. Output-path configuration also permits locations that
|
||||||
|
need reconciliation with the selected directory categories.
|
||||||
|
- Standard combined transcript paths already match the target. Layout creation
|
||||||
|
also creates an unused `transcripts/trimmed/` directory.
|
||||||
|
- Most subprocess diagnostics live under individual run directories. Root-level
|
||||||
|
log paths lack a consistent latest-attempt view. Audita reports sit beside
|
||||||
|
its staged transcript, and restore writes a separate top-level report.
|
||||||
|
- Prepared previous-session inputs occupy `previous/`. Narratio-owned reference
|
||||||
|
snapshots supplied to Notarius occupy session `cache/notarius/references/`.
|
||||||
|
These snapshots are distinct from Notarius's own execution caches.
|
||||||
|
- Audita debug placement follows downstream configuration. Narratio does not
|
||||||
|
expose Notarius debug capture, and Scriptorium render-debug placement varies.
|
||||||
|
- Narratio maintains separate S3 audio cache and spool paths in addition to the
|
||||||
|
prepared session audio copy.
|
||||||
|
|
||||||
|
## Target Layout And Ownership
|
||||||
|
|
||||||
|
Retain the configurable workspace root and its current system-wide default,
|
||||||
|
`/var/lib/narratio`. Sessions remain under
|
||||||
|
`/var/lib/narratio/work/<campaign>/<session>` by default. The following paths
|
||||||
|
express the selected categories:
|
||||||
|
|
||||||
|
| Path | Contents and role |
|
||||||
|
| --- | --- |
|
||||||
|
| `manifest.json` | Narratio's durable progress and output references. |
|
||||||
|
| `artifacts/<filename>` | Completed artifact payloads, including session bounds. |
|
||||||
|
| `transcripts/<filename>` | Completed combined transcripts and rendered transcripts. |
|
||||||
|
| `transcripts/raw/<speaker>.json` | Completed initial per-speaker transcripts. |
|
||||||
|
| `audio/` | Prepared external audio. |
|
||||||
|
| `inputs/` | Prepared configuration, references, and previous-session inputs. |
|
||||||
|
| `logs/<stage>` | Symlink to the latest attempt's run-local logs and execution reports. |
|
||||||
|
| `debug/<stage>` | Symlink to the latest attempt's run-local debug directory, empty when no capture is produced. |
|
||||||
|
| `runs/<run_id>/<stage>/logs/` | Actual logs and execution reports for that attempt. |
|
||||||
|
| `runs/<run_id>/<stage>/debug/` | Actual debug captures for that attempt, including retained state snapshots. |
|
||||||
|
| `runs/<run_id>/` | Narratio staging, original tool bundles, and retained execution evidence. |
|
||||||
|
|
||||||
|
For example, the latest views use relative links within the session:
|
||||||
|
|
||||||
|
```text
|
||||||
|
logs/polish -> ../runs/<run_id>/polish/logs/
|
||||||
|
debug/polish -> ../runs/<run_id>/polish/debug/
|
||||||
|
```
|
||||||
|
|
||||||
|
Each stage's latest attempt can belong to a different run. Producers write to
|
||||||
|
the actual run paths; the symlinks provide operator navigation without duplicate
|
||||||
|
diagnostic storage.
|
||||||
|
|
||||||
|
Narratio's lock and restore marker remain its own State. Their current placement
|
||||||
|
can remain; introducing another root-level State directory is unnecessary for
|
||||||
|
this feature. Generated records that describe an invocation belong with Logs;
|
||||||
|
configuration actually supplied as an input follows its input or staging role.
|
||||||
|
|
||||||
|
Subprocesses may create internal working directories wherever their configuration
|
||||||
|
directs. Narratio-selected staging destinations for result bundles remain valid
|
||||||
|
under ADR 0002. Cache locations are documented as operator recommendations using
|
||||||
|
each application's existing configuration surface.
|
||||||
|
|
||||||
|
## Required Changes
|
||||||
|
|
||||||
|
### Output paths and configuration
|
||||||
|
|
||||||
|
Centralize the revised paths in the existing path owner. Apply the flat artifact
|
||||||
|
rule to explicit artifacts, expanded families, extraction outputs, and session
|
||||||
|
bounds. Preserve logical source identities used by dependencies and publication.
|
||||||
|
Keep explicit analysis filenames and require flat destinations. Derive extraction
|
||||||
|
filenames from configured extraction keys with a `.json` extension, with an
|
||||||
|
explicit extraction filename override for naming preferences or collisions.
|
||||||
|
Retain `session_bounds.json` as the default bounds filename. Do not silently
|
||||||
|
rename conflicting files. Report conflicting producers and the shared destination
|
||||||
|
in configuration errors. Validate the effective configuration after composition
|
||||||
|
and family expansion.
|
||||||
|
|
||||||
|
Keep combined transcript destinations directly beneath `transcripts/`. Remove
|
||||||
|
obsolete directory scaffolding and reconcile configurable output paths with
|
||||||
|
their category. Local path changes do not require flattening explicitly
|
||||||
|
configured remote publish destinations.
|
||||||
|
|
||||||
|
### Notarius result handling and reuse
|
||||||
|
|
||||||
|
Receive and validate the complete Notarius bundle in Narratio staging. Promote
|
||||||
|
selected accepted payloads to their flat artifact destinations. Classify the
|
||||||
|
receipt, warnings, rejection summaries, and diagnostics as Logs. Keep bundle
|
||||||
|
indexes and other management metadata with the bundle or Narratio State according
|
||||||
|
to their use.
|
||||||
|
|
||||||
|
Revise extraction evidence, catalog hydration, and resume checks to resolve
|
||||||
|
promoted products independently of the old `artifacts/notarius/` layout. Retain
|
||||||
|
the identity, provenance, contract, and checksum evidence needed for current
|
||||||
|
reuse behavior. Historical bundles may remain available for inspection under
|
||||||
|
their existing retention semantics; relocating them beneath `runs/` must not
|
||||||
|
silently shorten their lifetime through run cleanup.
|
||||||
|
|
||||||
|
Preserve ordinary resume and forced-extraction behavior. Subprocess caches remain
|
||||||
|
under Notarius's control.
|
||||||
|
|
||||||
|
### Inputs
|
||||||
|
|
||||||
|
Place prepared previous-session material beneath `inputs/previous/`. Place the
|
||||||
|
stable reference snapshots Narratio supplies to Notarius beneath
|
||||||
|
`inputs/references/`, retaining their content identities where needed for reuse.
|
||||||
|
Update preparation, input resolution, manifest references, and restore mappings.
|
||||||
|
Keep source configuration ownership and prepared-file validation unchanged.
|
||||||
|
|
||||||
|
### Audio acquisition
|
||||||
|
|
||||||
|
Simplify acquisition to one prepared session copy under `audio/`. Remove the
|
||||||
|
separate Narratio S3 audio cache and routine retained spool copies, together with
|
||||||
|
their obsolete configuration, manifest bookkeeping, and cleanup controls.
|
||||||
|
Preparation may use temporary download files before installing validated audio.
|
||||||
|
Preserve reuse of valid prepared audio and update restore to use the same layout.
|
||||||
|
|
||||||
|
Recreating a session after its prepared audio is removed requires fetching the
|
||||||
|
source again. Original recordings remain outside this cleanup change. Removing
|
||||||
|
spool/cache handling does not authorize new deletion of prepared session audio.
|
||||||
|
Subprocess caches remain independently configured and managed.
|
||||||
|
|
||||||
|
### Logs and execution reports
|
||||||
|
|
||||||
|
Provide one consistent latest-attempt view per stage. Include failed attempts;
|
||||||
|
a skipped stage leaves its previous view intact. Distinguish multiple applications
|
||||||
|
or artifact invocations within a stage through their filenames. Route Seriatim
|
||||||
|
reports, Audita reports, Notarius diagnostic records, and Narratio invocation
|
||||||
|
records through this model. Treat restore execution as its own diagnostic group
|
||||||
|
under `logs/restore/`, using run-local evidence and a latest-attempt view.
|
||||||
|
|
||||||
|
Prevent a new attempt from presenting old diagnostic files as newly generated
|
||||||
|
ones. At the start of each actual stage attempt, create its run-local log and
|
||||||
|
debug directories and refresh both latest symlinks. An attempt without debug
|
||||||
|
output exposes an empty debug directory rather than an older capture. Preserve
|
||||||
|
subprocess errors and available evidence on failure. Retain existing
|
||||||
|
captured-stream redaction and limits.
|
||||||
|
|
||||||
|
Keep historical files at their run paths; rotating the links does not move files.
|
||||||
|
Run cleanup must preserve directories targeted by current latest links. Do not
|
||||||
|
introduce an automatic history age or count limit. Publication and manifest
|
||||||
|
diagnostic references use actual run paths. Adjust path handling for these
|
||||||
|
Narratio-managed navigation links without relying on arbitrary symlink traversal.
|
||||||
|
|
||||||
|
### Debug controls and placement
|
||||||
|
|
||||||
|
Add selective Narratio configuration for Audita and Notarius debug capture using
|
||||||
|
their supported enabling CLI flags. An enabled Narratio selector requests
|
||||||
|
capture; an omitted or false selector passes no debug override and preserves the
|
||||||
|
application's configured/default behavior. Do not add a downstream disable flag.
|
||||||
|
|
||||||
|
Supply the actual run-local debug directories, exposed through `debug/polish`
|
||||||
|
and `debug/extract`. Control the destination when capture can be enabled through
|
||||||
|
downstream configuration as well, including Audita overlays, using supported
|
||||||
|
destination controls without changing capture enablement. Audita's debug
|
||||||
|
work-directory option selects diagnostic storage and belongs to this placement
|
||||||
|
responsibility.
|
||||||
|
|
||||||
|
Route Scriptorium's existing analysis and bounds render-debug products to the
|
||||||
|
run-local directories exposed through `debug/analyze` and `debug/trim`. Preserve
|
||||||
|
its existing per-artifact selection capability. Seriatim requires no invented
|
||||||
|
debug mode; its ordinary reports are
|
||||||
|
Logs. Subprocess-defined bundle contents and nested run directories are allowed
|
||||||
|
beneath the selected run-local debug root. Logs and Debug share the same
|
||||||
|
latest-attempt semantics and cleanup protection.
|
||||||
|
|
||||||
|
Keep new full Audita and Notarius debug bundles local in this feature. Preserve
|
||||||
|
publication of currently declared diagnostics, including existing render-debug
|
||||||
|
products. Enabling capture does not implicitly add full debug bundles to remote
|
||||||
|
archives.
|
||||||
|
|
||||||
|
Debug controls govern capture when a stage executes. They do not independently
|
||||||
|
require regeneration of valid substantive outputs. Document the existing force
|
||||||
|
workflow for obtaining fresh diagnostic evidence when a stage would be reused.
|
||||||
|
Preserve downstream failure semantics for diagnostic writes and the existing
|
||||||
|
boundary of Narratio's captured-stream redaction.
|
||||||
|
|
||||||
|
### Audita destination support
|
||||||
|
|
||||||
|
Request upstream support for independently selecting the transcript output,
|
||||||
|
ordinary report/log directory, and debug/retained working directory, then adopt
|
||||||
|
a supporting release. Audita currently ignores `--report-dir` when transcript
|
||||||
|
output is a file; honoring that destination is the principal missing capability.
|
||||||
|
Narratio continues to capture subprocess stdout and stderr itself.
|
||||||
|
|
||||||
|
Direct ordinary reports, including failure reports, to the actual polish log
|
||||||
|
directory, the transcript to its staged product destination, and retained debug
|
||||||
|
material to the actual polish debug directory. This feature uses the supported
|
||||||
|
upstream controls in a clean cutover without version-based compatibility branches.
|
||||||
|
|
||||||
|
### Manifest, publication, restore, and cleanup
|
||||||
|
|
||||||
|
Update all consumers of changed paths and file categories together. Preserve
|
||||||
|
per-speaker and per-artifact completed-work reuse. Keep publication driven by
|
||||||
|
declared products and diagnostics, with new full debug bundles remaining local.
|
||||||
|
Restore new-format products and inputs to their new locations.
|
||||||
|
|
||||||
|
Define an explicit way to identify supported session state for the clean
|
||||||
|
cutover. Document regeneration with the new version and manual configuration
|
||||||
|
updates. Avoid accepting old state through path fallbacks or silently rewriting
|
||||||
|
old local sessions or remote snapshots.
|
||||||
|
|
||||||
|
Preserve existing invalidation and failed-replacement behavior. Review retention
|
||||||
|
by content role before changing cleanup targets, especially when durable products
|
||||||
|
or historical bundles move. Run cleanup preserves the current log and debug
|
||||||
|
link targets; explicit removal of an entire session removes its local views
|
||||||
|
with it. Subprocess working State and Cache remain outside Narratio's cleanup
|
||||||
|
responsibility.
|
||||||
|
|
||||||
|
### Documentation and verification
|
||||||
|
|
||||||
|
Update configuration examples, the operator guide, adapter contracts, workspace
|
||||||
|
internals, and the architecture policy when implementation lands. Replace the
|
||||||
|
current architectural assumption that extraction requires a complete immutable
|
||||||
|
bundle beneath `artifacts/` with the selected product-promotion model. Retire
|
||||||
|
policy and documentation for Narratio's removed audio cache/spool lifecycle and
|
||||||
|
document the managed latest-attempt symlinks. Keep per-application cache locations
|
||||||
|
as deployment recommendations owned by each application's configuration.
|
||||||
|
|
||||||
|
Use focused behavioral coverage for flat output placement, collision rejection,
|
||||||
|
latest-attempt diagnostics on success and failure, debug selection and routing,
|
||||||
|
unchanged completed-work reuse, failed replacement, and new-format publish/restore.
|
||||||
|
Cover empty debug views after attempts without capture, preserved views for
|
||||||
|
skipped stages, cleanup protection of latest targets, and audio acquisition,
|
||||||
|
reuse, and restore without persistent spool/cache duplicates. Exercise cleanup
|
||||||
|
against products and separately owned caches. Use temporary filesystems and
|
||||||
|
subprocess doubles; default tests remain offline and independent
|
||||||
|
of sibling checkouts or paid providers.
|
||||||
Reference in New Issue
Block a user