Plan the ConfigKit upgrade and workspace development

This commit is contained in:
2026-10-04 12:28:09 +00:00
parent 40cbf42438
commit fe9817f354
6 changed files with 1136 additions and 0 deletions

175
docs/roadmap/audita.md Normal file
View File

@@ -0,0 +1,175 @@
# Adopt Audita v3 and independent diagnostic destinations
## Purpose
Adopt Audita v3.0.0 for transcript polishing and use its independent output
destinations to implement the Audita portion of the
[session workspace roadmap](workspace.md). Operators should find the completed
transcript, ordinary reports, and optional debug evidence in their respective
Narratio locations.
This document defines the Audita adoption scope and target. The workspace
roadmap owns shared layout, latest-attempt views, and retention behavior. Its
referenced ADRs remain Proposed. Implementation sequencing belongs in a
subsequent implementation plan.
## Reviewed Upstream Release
The reviewed Audita v3.0.0 tag identifies commit
`3b4933256eff84327b812e7ae12babf743bc04a6`. Its
[release notes](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/releases/v3.0.0.md),
[CLI reference](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/cli.md),
[configuration reference](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/config.md),
and [operations guide](https://gitea.maximumdirect.net/eric/audita/src/tag/v3.0.0/docs/operations.md)
own the downstream contracts.
Audita now honors an explicit `--report-dir` alongside file transcript output.
Its existing `--work-dir` selects the parent of retained debug captures, with
capture enabled separately through configuration or `--debug`. These controls
satisfy Narratio's requested destination separation. Further upstream changes
are unnecessary for this scope.
## Current Baseline
Narratio's [Audita integration](../integrations/audita.md) and
[polish stage](../internal/stage-polish.md) describe the implemented behavior:
- The adapter invokes `process` with a transcript, glossary, and staged transcript
destination. It supports independent downstream configuration, overlay,
profile, module, description, and output-schema selectors.
- The adapter supplies no reporting, debug, or work-directory flags. Audita
controls debug capture and placement through its own configuration.
- Ordinary reports are discovered beside the staged transcript. The polish
stage validates returned report paths against those adjacent filenames.
- Only the validated polished transcript becomes a canonical stage output.
Reports, captured streams, and the generated invocation record are diagnostics.
- Selected credential references use Narratio's shared exact-name forwarding and
captured-stream redaction boundary.
The existing invocation remains usable with Audita v3 after deployment-side
profile migration. Narratio changes are needed to request and consume the
new destination separation.
## Targeted End State
### Products and diagnostics
Narratio supplies separate destinations for each actual polish attempt:
| Content | Destination and control |
| --- | --- |
| Corrected transcript | Run-local staging via `--output`; validated and promoted to `transcripts/polished.json` on success. |
| Ordinary reports | `runs/<run_id>/polish/logs/` via `--report-dir`. |
| Captured stdout, stderr, and Narratio invocation record | The same run-local polish log directory. |
| Optional debug evidence | `runs/<run_id>/polish/debug/` via `--work-dir`; Audita may create its own `run-*` subdirectory beneath it. |
The workspace roadmap's `logs/polish` and `debug/polish` links expose the latest
attempt. Narratio supplies actual run destinations to Audita and records those
paths in manifest diagnostics. Shared link updates and cleanup protection are
implemented by the workspace work rather than separately inside the adapter.
### Capture selection
Add an optional `pipeline.audita.debug` boolean, defaulting to false. A true
value passes `--debug`. False or omission supplies no capture override and
preserves Audita configuration and overlay behavior; it does not disable debug
enabled downstream.
Supply `--work-dir` for each executing attempt so downstream-enabled capture
also lands in the Narratio-selected debug directory. Selecting that destination
alone does not enable capture. Ordinary production operation retains Audita's
default reporting and debug settings. Narratio adds no reporting enable/disable
flag or debug-disable flag.
Debug selection affects diagnostics for an executing stage. Changing it alone
does not invalidate a completed transcript. Operators use the existing force
workflow when fresh polish or debug evidence is needed.
### Ownership and compatibility
Use a clean adoption of Audita v3's CLI contract, without version-dependent
adapter branches or legacy profile conversion. Audita continues to own its
configuration, profiles, prompts, runtime State, and diagnostic contents.
Narratio owns supplied destinations and transcript promotion.
Keep existing selectors and shared credentials such as `OPENROUTER_API_KEY` and
`RAKESTRAWHOME_API_KEY` under their configured exact names. Audita's examples
do not require Narratio to adopt application-specific credential aliases.
Operators continue to list the credential references required by all configured
Audita backends.
Audita is a separately installed executable. Its adoption of ConfigKit v0.5.0
and PromptKit v0.12.0 does not require changing Narratio's own library
dependencies as part of this feature.
## Required Changes
### Deployment configuration migration
Document the manual migration of Audita-owned PromptKit files:
- Wrap profile definitions under keyed `profiles:` mappings, remove inner `id`
fields, and replace `base_profile` with `extends`. Preserve selected profile
IDs when appropriate.
- Review explicit zeros, omitted values, and nulls against the new generation
control semantics. Validate the final configuration root, including unused
definitions and backend references.
- Rename any backend literally named `extends` and update its references.
- Validate production configuration and each used testing overlay with Audita's
offline `config validate` command. Audita application configuration continues
to use `version: 2`.
Keep the detailed downstream format owned by the pinned Audita documentation.
Narratio does not inspect, rewrite, or automatically migrate these files.
Deployment can migrate Audita before the workspace changes land.
### Adapter and polish stage
Carry explicit report and debug destinations through the Audita request and
construct the supported flags inside the adapter. Wire the debug selector
through configuration and runner composition.
Discover the existing three report filenames in the requested report directory.
Update polish's expected-path validation to the same location. Update invocation
records and safe result metadata to describe the selected destinations and
capture request. Preserve available failure evidence and the original subprocess
error.
Keep existing transcript validation, report read bounds, timeouts, credential
forwarding, and captured-stream redaction. Reports remain optional diagnostics;
early failures or downstream-disabled reporting may produce none. Exit-zero
runs with recoverable model errors remain successful. A nonzero exit, including
a late report-write failure, never authorizes transcript promotion. Narratio
does not add report-schema parsing or report-status success gates.
### Workspace integration and lifecycle
Use the shared workspace path owner for run-local log/debug destinations and
the agreed latest-attempt behavior. Preserve previous views on skipped stages;
actual attempts refresh both views, including on failure. An attempt without
debug capture exposes an empty current debug directory.
Keep full Audita debug bundles local in this feature. Preserve existing declared
diagnostic publication and apply the workspace roadmap's retention protection
to latest link targets. This feature inherits that roadmap's clean session
cutover and unchanged treatment of older completed outputs after invalidation
or failed replacement.
### Documentation and verification
When implemented, update the configuration reference and examples, Audita
integration contract, polish internals, and operator guide. Reference Audita v3's
profile migration and explain the shared Audita/Notarius overlay development
workflow. Downstream configuration edits behind unchanged selectors still
require explicitly forcing the affected stage to replace reused outputs.
Revise the workspace roadmap's upstream-request description when this adoption
is incorporated; Audita v3.0.0 supplies the previously missing capability.
Use focused offline tests for separate transcript/report/debug destinations,
enable-only debug selection, capture enabled by downstream configuration,
available failure reports, and preservation of nonzero-exit behavior. Verify
that reports remain diagnostics and only a successful validated transcript is
promoted. Shared latest-view and cleanup tests belong to the workspace change.
Default Narratio tests use subprocess doubles and temporary filesystems and
remain independent of sibling checkouts, real credentials, and paid providers.

202
docs/roadmap/config.md Normal file
View File

@@ -0,0 +1,202 @@
# Adopt ConfigKit v0.5.0
## Purpose
Upgrade Narratio from ConfigKit v0.4.2 to v0.5.0 and integrate its native
sibling inheritance with Narratio's configuration validation and inspection.
Keep the upgrade focused on the existing loading model and the adjustments
needed to consume the new release correctly.
This adoption precedes [Audita adoption](audita.md) and the
[workspace reorganization](workspace.md). This document defines the target
and required changes; [the implementation plan](implementation.md) defines
the steps for reaching that target.
## Reviewed Upstream Release
ConfigKit v0.5.0 identifies commit
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. The
[release notes](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/releases/v0.5.0.md),
[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/config.md),
and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/consumers/README.md)
own the upstream behavior.
The release introduces native `extends` relationships between sibling mappings,
layered loading with ordered underlays, caller-owned filesystem and mapping
sources, and passive source-resource contexts. Inheritance preserves explicit
presence, provenance, and secret classification.
The existing loading API remains available. ConfigKit's minimum Go line, YAML
dependency, and LGPL-3.0-or-later license are unchanged. Narratio can retain its
current Go toolchain and ordinary loading calls.
## Current Baseline
The [configuration reference](../config.md) owns Narratio's current public
semantics; [configuration internals](../internal/configuration.md) describe
loading and inspection.
- Pipeline loading uses ConfigKit with explicit additive imports and one
selected overlay, typed defaults, recorded resolution, and domain validation.
- Campaign, session, and canonical-party inputs use separate single-source
loads. Secret-directory bootstrap remains an explicit invocation operation.
- Fields with root-relative pipeline path semantics retain that base when their
declarations move into imports. For example, enabled Notarius configuration
and working-directory paths resolve from the root pipeline directory.
- Narratio adds a declaration-ownership check: each artifact or artifact-family
key may be declared by only one base file. Overlay modifications are allowed.
- Source inspection and artifact catalogs consume retained ConfigKit metadata.
Effective show, diff, and digest projections describe the resolved domain
configuration rather than source layout.
The review selected v0.5.0 through a temporary module file and ran Narratio's
existing test suite successfully. An additional probe exposed an integration
gap: a child artifact inheriting from a parent in another imported file is
rejected as declared by multiple base files. The same definitions in one file
load successfully. The check currently counts inherited history contributions
as if they were direct declarations.
## Targeted End State
### Native inheritance
Use ConfigKit's built-in sibling inheritance wherever its input contract and
Narratio's domain schema permit it. ConfigKit resolves `extends` after source
composition and overlay precedence, before typed decoding and application
resolution. Narratio receives the resulting typed configuration.
Inheritance can reduce repetition between artifacts and artifact families,
including definitions in separate imported files. Distinct child keys retain
their identities. Each resulting entry undergoes ordinary Narratio validation,
family expansion, dependency resolution, and execution selection.
Parent entries remain ordinary domain entries. They must satisfy applicable
validation and participate in the effective configuration. This adoption does
not introduce abstract artifacts or a separate template registry. Operators
must override inherited fields where needed to keep the resulting configuration
valid, such as output destinations.
Inherited false, zero, empty, and null input retains ConfigKit's explicit
presence semantics. Narratio's absence-based defaults must not refill such
values. Normal domain validation still determines whether a resulting value
is acceptable.
### Declaration ownership and provenance
Distinguish direct base declarations from inherited contributions. A child
declared in one file may inherit from a separately declared parent in another
file without becoming a duplicate declaration.
Retain the existing rejection of one artifact or family key directly declared
by multiple base files, including disjoint contributions and declarations
later removed by an overlay. Inheritance is not permission to merge multiple
direct declarations of the same key.
Keep defining-file ownership distinct from value-contributor provenance.
Existing source and artifact inspection must describe inherited contributors
accurately without presenting them as conflicting declarations. ConfigKit's
retained metadata remains the source of that evidence.
For generated family members, the family's defining source identifies the
child family declaration; inherited parent files remain contributor evidence.
Keep existing logical artifact IDs and inspection interfaces.
Effective configuration, diffs, and digests continue to reflect resolved
domain values. Equivalent inherited and explicitly authored configurations
should have equivalent effective projections, regardless of their provenance.
Stage reuse continues to use existing semantic evidence.
### Loading and ownership boundaries
Retain ordinary ConfigKit loading, centralized typed defaults, imports, selected
overlays, domain resolution, credential bootstrap, and exact-name credential
forwarding. Preserve current path bases and read-only inspection workflows.
Layered loading, embedded configuration sources, programmatic mapping sources,
and source-relative resource rebasing are available upstream but are outside
this focused adoption. Narratio continues to own its domain rules; ConfigKit
owns structural inheritance, presence, provenance, and secret resolution.
## Required Changes
### Dependency and attribution
Update `go.mod` and `go.sum` to ConfigKit v0.5.0. Update its entry in
[attribution](../oss/attribution.md) to the selected version and the reviewed
commit's source and notice links. Existing applicable license texts can remain.
### Configuration integration
Adjust the existing duplicate-definition check to identify direct declarations
using public ConfigKit history and origin metadata. Preserve genuine duplicate
detection while excluding structural transfers from declaration counting.
Use the completed load's retained evidence rather than reparsing files or
performing a second load that could resolve superseded credential references.
Review source ownership and artifact-definition projections against the new
inheritance metadata. Update only where needed to report contributors correctly.
Keep effective rendering and digest construction grounded in the existing safe
typed projection, with credential identities rather than resolved values.
Use ConfigKit's native operation directly. No application-owned inheritance
engine, collection registration, alternative keyword, or compatibility loader
is needed.
### Compatibility and operator guidance
Document that `extends` is now reserved structural input throughout ConfigKit
loads, including dynamic mappings such as artifact IDs and prompt variables.
Operators using that key as ordinary data must rename it. Inheritance is active
without an opt-in switch; imports and overlay-file selection remain separately
controlled as before.
Explain native sibling lookup, inherited explicit presence, and the requirement
that parent and child entries satisfy Narratio's domain rules. Link to the
pinned upstream contract for detailed inheritance semantics. Clarify that a
selected overlay may contribute sibling inheritance directives; this does not
add inheritance or stacking between overlay files.
Update the configuration reference, internal configuration guide, and a small
maintained example when implementation lands. Preserve each field's current
path-resolution rule when its value is inherited. Record the reserved-key
compatibility impact in the eventual release note; automatic configuration
conversion is outside this feature.
### Verification
Add focused Narratio coverage for the confirmed cross-file artifact inheritance
gap and the corresponding artifact-family behavior. Retain coverage for real
duplicate declarations, including those hidden by overlay replacement or
removal. Verify inherited explicit values against Narratio's defaulting rules,
credential safety, contributor reporting, and equivalent effective projections.
Concentrate tests on Narratio's integration behavior. ConfigKit's own tests own
its generic inheritance algorithm and source lifecycle. Narratio's default
tests remain offline, use temporary configuration bundles and controlled secret
values, and do not depend on the sibling ConfigKit checkout. Run the existing
suite against the selected released module before completing the upgrade.
## Completion Criteria
Narratio selects the released ConfigKit v0.5.0 module without a local checkout
replacement. Cross-file artifact and family inheritance works through the
ordinary loader, while genuinely duplicated base declarations remain invalid.
Inspection distinguishes declaration ownership from inherited contributors;
effective projections and credential handling retain their existing semantics.
Maintained guidance and examples explain native inheritance and the reserved
`extends` key. Attribution identifies the selected upstream revision. Focused
regression coverage and the existing suite pass without implementing Audita or
workspace changes as part of this upgrade.
## Relationship To Subsequent Work
ConfigKit adoption provides a verified configuration baseline for the later
Audita and workspace features. It makes no changes to subprocess destinations,
session layout, or cleanup behavior.
Audita adoption then migrates downstream profiles and adds adapter support for
explicit report/debug destinations. Audita owns its separate ConfigKit and
PromptKit dependencies. Shared latest-attempt links, final workspace paths, and
retention protection remain part of the workspace reorganization, so those
facilities are implemented once at their shared owner.

View File

@@ -0,0 +1,319 @@
# ConfigKit v0.5.0 implementation plan
## Scope and authority
Implement [the ConfigKit roadmap](config.md). It owns the target, compatibility
choices, and rationale. This plan covers Narratio's upgrade from v0.4.2 to
v0.5.0 and integration with native sibling inheritance. Audita adoption and
workspace reorganization remain subsequent, separately planned work.
The reviewed upstream revision is
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`, tagged `v0.5.0`.
Use the released module, not a local replacement. The pinned
[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/config.md)
and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/dc09959cca4e105f49ec316c6d4ba585bd2340bd/docs/consumers/README.md)
own upstream semantics. Read their sibling-inheritance and metadata sections
when working on the relevant stages; source may also be inspected in the
selected Go module cache. The sibling checkout is optional reference material,
not a build or test dependency.
## Working instructions
The audience is a Luna coding agent. Complete the numbered stages in order.
Each stage is a bounded unit with its own verification and completion criteria.
Inspect current code before editing; the paths below identify owners rather
than requiring changes to every listed file.
- Start with `docs/development.md`, `docs/policy/agents.md`, and `config.md` in
this directory. Follow the development policy-reading guide. Architecture,
testing, documentation, and OSS policies apply to this work; release policy
applies if discussing compatibility or eventual release preparation.
- Use Go 1.26. Prefer the repository's graph tools for code discovery and
inspect focused tests at each package boundary.
- Keep ordinary `Load`, typed defaults, explicit imports, one selected overlay,
domain validation, and current path-resolution rules. Native inheritance is
available wherever ConfigKit and Narratio's schema permit it, with no new
collection allowlist, opt-in flag, or Narratio-owned inheritance engine.
- Distinguish original declarations from inherited contributors. Preserve
genuine duplicate-base rejection and truthful provenance. Parent entries
remain ordinary entries subject to existing domain rules.
- Keep invocation-owned secrets bootstrap and shared exact-name credential
forwarding unchanged. Do not introduce a second base load, file reparse, or
historical-secret lookup to recover declaration information.
- Tests use temporary files, controlled synthetic secrets, and real internal
collaborators. No downstream executable, sibling checkout, cloud service,
or paid provider is needed. Protect integration behavior rather than
duplicating ConfigKit's generic algorithm tests or freezing incidental output.
- Commit each completed nonempty unit with a short plain-English message after
suitable validation. Report blockers and incomplete work accurately. If a
review-only task needs no source fix, retain useful behavioral coverage and
report that result rather than making a cosmetic rewrite.
Do not adopt layered loading, new source types, source-relative rebasing,
abstract artifacts, automatic YAML conversion, new execution selection rules,
new output collision policy, or changes to subprocess credentials and workspace
paths in this feature. Existing domain validation and reuse rules still apply;
do not add prohibitions, hardening, or guarantees merely because they can be
tested. This plan does not authorize a release, push, tag, ADR acceptance, or
retirement of pending roadmaps.
## Stage 1: Select the released dependency and update attribution
Purpose: establish the actual dependency baseline before integration work.
Read `go.mod`, `go.sum`, `docs/oss/attribution.md`, the OSS policy, and the
selected release's notes, `COPYRIGHT`, and `LICENSE`.
1. Select `gitea.maximumdirect.net/eric/ConfigKit@v0.5.0` with the normal Go
module workflow. Keep Go 1.26 and avoid unrelated dependency upgrades.
2. Update ConfigKit's attribution entry to v0.5.0 and use the full reviewed
commit for its Source, COPYRIGHT, and LICENSE links. Preserve upstream
copyright wording, SPDX expression, and applicable local license links.
Its license and YAML dependency are unchanged; no new license copy is needed.
3. Inspect module changes for unintended churn or a local `replace`. Existing
configuration loaders should compile without adopting optional APIs.
Verify with `go test ./internal/config ./internal/app ./internal/doccheck`
and `go mod tidy -diff`. Confirm
`go list -m gitea.maximumdirect.net/eric/ConfigKit` selects v0.5.0 without a
replacement.
Complete when the released dependency and notices agree and current focused
tests pass. The cross-file inheritance defect is addressed in the next stage;
this stage does not claim that new inheritance behavior is fully integrated.
## Stage 2: Correct direct declaration accounting
Purpose: allow distinct artifact and family keys to inherit across imported
files while preserving existing duplicate-declaration rejection.
Read `internal/config/pipeline_configkit.go`, particularly
`pipelineMetadataFromConfigKit` and `rejectDuplicateBaseDefinitions`, and
`internal/config/pipeline_configkit_test.go`. Inspect the public ConfigKit
`Metadata.History`, `HistoricalPaths`, `Origin`, and `ChangeKind` contracts.
1. Add package-boundary regression fixtures with a valid parent in one imported
file and a child using `extends` in another. Cover ordinary artifacts and
artifact families. Give parent and child distinct valid output paths or
family patterns, and provide canonical-party context for expanded families.
Both keys remain ordinary domain entries; child IDs must stay their own.
2. Adjust declaration accounting using retained public metadata. In v0.5.0,
original base-input events have `LayerBase` and no origin operation;
projected transfers carry `ChangeInheritance`. Count original base
declarations, not these transfers or subsequent decoding/resolution events.
Consumed structural `extends` paths are not artifact declarations.
3. Keep the historical-path check, including original declarations erased by
a selected overlay. Do not switch to surviving effective keys only or count
all displaced origins as fresh declarations. Reuse a small package-local
helper only if it makes the next stage clearer.
4. Retain `TestPipelineConfigKitRejectsDuplicateBaseDeclarations`, including
disjoint fields supplied to the same key, container removal, and removal of
Scriptorium by an overlay. An overlay may still modify an existing entry.
Add only any missing regression case needed to exercise the new filter.
Verify with `go test ./internal/config`. The positive cross-file regression
should expose the baseline defect before the fix. Assert loaded values and
identities, or the duplicate category and relevant sources, without matching an
entire diagnostic or ConfigKit's private event ordering.
Complete when cross-file inheritance loads and expands correctly, and actual
duplicates still fail even when the overlay hides them. Do not use the direct
declaration filter to remove contributor evidence from inspection.
## Stage 3: Preserve contributor provenance and family ownership
Purpose: keep inspection useful when one entry contains values from several
files, without confusing its defining file with those contributors.
Read `internal/config/artifact_catalog.go`,
`internal/config/artifact_families.go`, `internal/config/effective_pipeline.go`,
and metadata-to-ownership projection in `pipeline_configkit.go`. Inspect the
focused catalog, family, and effective-source tests.
1. Exercise the inherited configurations from Stage 2 through
`EffectiveArtifactCatalog` and `EffectivePipelineSources`. Check inherited
values against their actual parent contributors and child overrides against
the child/overlay contributors. Keep historical definition evidence distinct
from effective field ownership.
2. Preserve `DefinitionSources` aggregation, including inherited contributors.
Do not relabel all inherited values as authored in the child's file.
3. Correct `artifactFamilySource` where its first inherited field currently
selects the parent's file. For an explicitly declared child family, identify
its original declaration using retained metadata: use its base defining file
when present, or its overlay defining file for an overlay-only family.
Overlay edits to a base family do not relocate its base definition.
4. Retain the existing fallback for entries without a direct declaration or
programmatically constructed configuration. Do not restrict valid enclosing
inheritance merely to force every entry into a single-file ownership model.
5. Keep logical `SourceID`, artifact keys, family/character/party provenance,
existing source roles, and inspection interfaces. Update projections only
where inherited evidence is otherwise lost or incorrectly attributed.
Verify with `go test ./internal/config`. Cover an explicitly declared child
family inheriting from another file and a selected-overlay contribution. Check
meaningful records and source identities, not a complete formatted snapshot.
Complete when contributor reporting remains truthful and generated members
identify their child family definition rather than an arbitrary inherited field.
## Stage 4: Verify inherited presence, domain resolution, and path bases
Purpose: prevent Narratio's application defaults or normalization from undoing
the library's inherited explicit input.
Read `pipelineConfigDefaults`, `resolvePipelineConfig`, and
`internal/config/load.go`, including `finalizeLoadedPipeline` and
`resolveNotariusPaths`. Inspect the current presence/defaulting tests and
the relevant domain validators.
1. Add representative inheritance cases for explicit false, zero, empty,
and null input versus genuinely absent input. Include a pointer boolean
and an entry-level absence-based default where Narratio consumes metadata.
For example, inherited `render_debug: false` can exercise artifact presence;
inherited empty Scriptorium timeout can exercise an existing invalid value
that must not silently become its default.
2. Preserve the existing domain outcome for each chosen field. Valid explicit
values retain their value; explicit values already invalid must still fail
ordinary validation rather than be refilled. An empty mapping merges under
ConfigKit's rules and must not be described as universal container removal.
3. Exercise overlay composition before inheritance: modify a parent's field
in the selected overlay and confirm the child's inherited result uses that
effective value. A child override still wins over its parent.
4. Verify a known root-relative field, such as enabled Notarius `config_path`
or `working_directory`, when inherited from an imported declaration.
Resolve it exactly as the existing field contract does. Do not use passive
SourceContext to rebase it to the parent's or child's source directory, or
newly normalize other path fields merely to make their behavior uniform.
5. Fix any demonstrated Narratio integration defect at its existing defaulting
or domain-resolution owner. Use ConfigKit presence and recorded operations;
do not recreate structural inheritance or weaken ordinary validation.
Verify with `go test ./internal/config`. Keep coverage representative; generic
missing-parent, cycle, resource-context, and expansion-limit matrices belong to
ConfigKit. Parents must remain valid Narratio entries, not special templates.
Complete when inherited explicit input and absent defaults have distinct,
correct domain outcomes and existing path bases are unchanged.
## Stage 5: Verify safe effective projections and inherited credentials
Purpose: ensure inheritance changes neither semantic comparison nor the
existing secret boundary.
Read `internal/config/effective_pipeline.go`, `effective_digest.go`,
`effective_projection.go`, `scriptorium_credentials_test.go`, and the effective
projection tests in `load_validate_test.go`. For diff composition, read
`internal/app/config_diff.go`. Reuse
`TestPipelineOverlayDoesNotRequireUnusedBaseCredentials` and
`TestCredentialValuesNeverEnterEffectiveProjectionOrDigest` coverage.
1. Load two configurations that produce the same effective domain values:
one uses inheritance, the other writes the expanded values explicitly.
Include ordinary artifacts and representative resolved family members.
Verify equal safe effective mappings and digests and no effective diff.
Do not compare private provenance as part of semantic equality.
2. Add a representative typed credential inheritance case using shared fields
of valid Narratio sibling mappings and controlled synthetic environment
variables. Verify resolved credentials retain their exact configured names;
serialization and inspection expose those identities without values.
3. Verify overlay removal/replacement of a credential before inheritance does
not resolve the superseded reference. Reuse the existing selected-overlay
regression rather than creating a second loader or bootstrap algorithm.
4. Check that changing only synthetic secret values leaves semantic projections
and digest unchanged. Fix only demonstrated projection defects; preserve
the existing safe typed rendering boundary and semantic reuse evidence.
Verify with `go test ./internal/config ./internal/app`. Existing shared adapter
credential-forwarding tests remain the owner of subprocess forwarding; do not
rewrite adapters or duplicate those tests simply because input can be inherited.
Complete when resolved values, not source layout or secret bytes, determine
effective comparison, and inherited credentials obey the existing safe boundary.
## Stage 6: Document inheritance and maintain a small example
Purpose: make the adopted behavior understandable and usable without expanding
Narratio's configuration surface.
Read `docs/config.md`, `docs/internal/configuration.md`, `examples/README.md`,
the production/testing bundle, and `TestExamplesLoadAndValidate` in
`internal/config/load_validate_test.go`. Follow the documentation policy.
1. Add native sibling inheritance guidance to the configuration reference.
Explain exact sibling lookup, overlay-before-inheritance resolution,
explicit presence, ordinary parent validation, distinct child identity,
and cross-file declaration/contributor distinctions. Link to the pinned
upstream contract for detailed syntax and structural rules.
2. Explain the reserved `extends` key throughout ConfigKit inputs, including
dynamic artifact IDs and prompt-variable mappings. Operators must rename
existing ordinary-data keys with that name; there is no opt-in or automatic
conversion. Campaign, session, and party inputs also use ConfigKit.
3. Clarify the existing statement that overlays cannot inherit: overlay files
cannot inherit or stack, but the selected overlay may supply native sibling
directives within its values. Keep imports and overlay selection unchanged.
4. Update configuration internals for inheritance before typed decoding,
original-declaration counting, contributor projection, and the safe
effective boundary. Describe implemented mechanisms, linking to the public
reference for their operator-facing meaning.
5. In `examples/production-testing/overlays/testing.yml`, add
`extends: session_recap` to the existing `testing_notes` artifact. Keep its
disabled status and distinct output override. It inherits the imported
parent's prompt/input and the parent's testing profile after overlay
application. The parent remains an ordinary artifact.
6. Update the example guide and extend existing example validation to check
those inherited fields. Preserve the bundle's current identities and
artifact count; no new example root or template convention is necessary.
Verify with `go test ./internal/config -run '^TestExamplesLoadAndValidate$'`
and `go test ./internal/doccheck`. Review examples for synthetic-only content
and validate local links. Keep unimplemented Audita/workspace behavior in its
roadmaps. Do not create a release note under an invented Narratio version;
reserved-key compatibility impact remains an explicit release-preparation item.
Complete when canonical documentation and a tested copyable example describe
the implemented adoption, without promoting new restrictions to policy.
## Stage 7: Verify the assembled inspection workflow and close the feature
Purpose: establish that real command composition consumes the integrated
configuration correctly and finish the upgrade as a reviewable baseline.
Read `internal/app/config_commands_test.go`, `config_artifacts_test.go`,
and existing read-only inspection workflow coverage. Reuse the preceding
package coverage rather than repeating its entire matrix at the CLI boundary.
1. Add or extend a representative command-level temporary imported bundle with
inheritance. Exercise validation, effective show, source reporting, artifact
catalog, and diff through existing command entry points. Use a campaign/party
fixture for family inspection, and paired overlays with equivalent inherited
and explicitly expanded values when testing command-level diff equality.
2. Check semantic output: resolved child values, expected contributor/defining
sources, logical artifact identities, and no secret values. Keep inspection
read-only with respect to session/runtime state, as existing tests establish.
Do not invoke downstream applications or add CLI modes or columns.
3. Review the combined diff against `config.md` and this plan. Confirm that
ConfigKit v0.5.0 is selected, attribution links use the reviewed revision,
no optional loading feature was adopted, and no Audita/workspace change or
compatibility shim slipped into scope.
4. Run `go test ./...`, `go test -race ./...`, `go vet ./...`,
`go build ./...`, `go mod tidy -diff`, and `git diff --check` against the
final source. These reuse existing repository checks, not new release gates.
The full suite includes example and documentation-link validation; focused
checks need not be repeated solely for this stage if unchanged.
5. Commit the completed unit and report the dependency revision, checks,
compatibility impact, and any unfinished work. Leave the pending Audita and
workspace plans and Proposed ADRs intact. Retire completed planning files
only during separately requested cleanup/release preparation, repairing links.
Complete when the feature roadmap's completion criteria are evidenced by the
released dependency, integrated behavior, maintained documentation/examples,
and passing final checks. The next implementation scope is Audita; its concrete
plan is prepared separately. Release version selection and publication are
outside this implementation pass.
## Open Questions
None required for this adoption. Existing decisions select native inheritance,
current loading/path/credential semantics, and ConfigKit before Audita before
workspace work. Eventual release preparation must account for the reserved-key
compatibility impact under the release policy; no version is selected here.

280
docs/roadmap/workspace.md Normal file
View File

@@ -0,0 +1,280 @@
# Session workspace organization
## Purpose
Give operators predictable locations for session inputs, completed outputs,
logs, and debug evidence. Apply the taxonomy in
[ADR 0001](../adr/0001-classify-session-content.md) and the ownership boundary in
[ADR 0002](../adr/0002-assign-subprocess-content-ownership.md).
This roadmap defines the target behavior and required changes. Implementation
sequencing belongs in a subsequent implementation plan. The ADRs remain
Proposed pending their acceptance.
## Confirmed Requirements
- Completed artifacts occupy a flat `artifacts/` directory. Intermediate
products such as `session_bounds.json` belong here.
- Combined transcripts occupy `transcripts/` directly. Initial per-speaker
transcripts occupy a subdirectory beneath it.
- Prepared external audio occupies `audio/`; other prepared external inputs
occupy `inputs/`. Outputs consumed by later stages retain their output paths.
- Logs and Debug use matching latest-attempt views: `logs/<stage>` and
`debug/<stage>` are symlinks to the producing run's directories. Both views
advance when a stage starts an actual attempt, including attempts that fail.
A skipped stage leaves its previous views intact.
- Execution reports and invocation records are Logs. Historical Logs and Debug
remain under their producing runs, subject to cleanup.
- Narratio can selectively request supported subprocess debug capture through
configuration. Omission preserves application configuration and defaults;
Narratio needs no explicit downstream debug-disable override.
- Prepared session audio is the single retained Narratio-managed audio copy.
Remove Narratio's separate persistent audio cache and spool duplicates.
- Configurations that produce colliding artifact names or destinations fail
validation before execution. Validation covers the effective pipeline,
including expanded character families and built-in artifact destinations.
- Narratio selects input, output, log, and debug paths. Subprocess applications
own their working State and Cache through their supported configuration.
- A dedicated cache root such as `/var/cache/notarius` is a deployment
recommendation. The application owns its cache defaults, internal layout,
identity, and reuse semantics. Narratio session cleanup respects that boundary.
- Narratio retains its own manifest, recovery records, and managed staging State.
- This is a clean cutover for newly prepared sessions. Existing-session migration
and backward compatibility shims are outside the feature.
- Treatment of older completed files after invalidation or failed replacement
remains unchanged, including their availability through manifest-backed
resolution. Directory reorganization does not authorize new stale-file
deletion, automatic fallback, or rollback behavior.
## Current Baseline
The [operator guide](../operations.md#local-state-layout) owns the current
physical layout. Relevant differences from the target are:
- Extraction moves the complete Notarius bundle to
`artifacts/notarius/<run_id>/`. Resume evidence and artifact resolution depend
on that bundle and its index.
- Merge and normalize promote Seriatim execution reports into `artifacts/`.
- Analysis output paths and character-family patterns can introduce nested
artifact directories. Output-path configuration also permits locations that
need reconciliation with the selected directory categories.
- Standard combined transcript paths already match the target. Layout creation
also creates an unused `transcripts/trimmed/` directory.
- Most subprocess diagnostics live under individual run directories. Root-level
log paths lack a consistent latest-attempt view. Audita reports sit beside
its staged transcript, and restore writes a separate top-level report.
- Prepared previous-session inputs occupy `previous/`. Narratio-owned reference
snapshots supplied to Notarius occupy session `cache/notarius/references/`.
These snapshots are distinct from Notarius's own execution caches.
- Audita debug placement follows downstream configuration. Narratio does not
expose Notarius debug capture, and Scriptorium render-debug placement varies.
- Narratio maintains separate S3 audio cache and spool paths in addition to the
prepared session audio copy.
## Target Layout And Ownership
Retain the configurable workspace root and its current system-wide default,
`/var/lib/narratio`. Sessions remain under
`/var/lib/narratio/work/<campaign>/<session>` by default. The following paths
express the selected categories:
| Path | Contents and role |
| --- | --- |
| `manifest.json` | Narratio's durable progress and output references. |
| `artifacts/<filename>` | Completed artifact payloads, including session bounds. |
| `transcripts/<filename>` | Completed combined transcripts and rendered transcripts. |
| `transcripts/raw/<speaker>.json` | Completed initial per-speaker transcripts. |
| `audio/` | Prepared external audio. |
| `inputs/` | Prepared configuration, references, and previous-session inputs. |
| `logs/<stage>` | Symlink to the latest attempt's run-local logs and execution reports. |
| `debug/<stage>` | Symlink to the latest attempt's run-local debug directory, empty when no capture is produced. |
| `runs/<run_id>/<stage>/logs/` | Actual logs and execution reports for that attempt. |
| `runs/<run_id>/<stage>/debug/` | Actual debug captures for that attempt, including retained state snapshots. |
| `runs/<run_id>/` | Narratio staging, original tool bundles, and retained execution evidence. |
For example, the latest views use relative links within the session:
```text
logs/polish -> ../runs/<run_id>/polish/logs/
debug/polish -> ../runs/<run_id>/polish/debug/
```
Each stage's latest attempt can belong to a different run. Producers write to
the actual run paths; the symlinks provide operator navigation without duplicate
diagnostic storage.
Narratio's lock and restore marker remain its own State. Their current placement
can remain; introducing another root-level State directory is unnecessary for
this feature. Generated records that describe an invocation belong with Logs;
configuration actually supplied as an input follows its input or staging role.
Subprocesses may create internal working directories wherever their configuration
directs. Narratio-selected staging destinations for result bundles remain valid
under ADR 0002. Cache locations are documented as operator recommendations using
each application's existing configuration surface.
## Required Changes
### Output paths and configuration
Centralize the revised paths in the existing path owner. Apply the flat artifact
rule to explicit artifacts, expanded families, extraction outputs, and session
bounds. Preserve logical source identities used by dependencies and publication.
Keep explicit analysis filenames and require flat destinations. Derive extraction
filenames from configured extraction keys with a `.json` extension, with an
explicit extraction filename override for naming preferences or collisions.
Retain `session_bounds.json` as the default bounds filename. Do not silently
rename conflicting files. Report conflicting producers and the shared destination
in configuration errors. Validate the effective configuration after composition
and family expansion.
Keep combined transcript destinations directly beneath `transcripts/`. Remove
obsolete directory scaffolding and reconcile configurable output paths with
their category. Local path changes do not require flattening explicitly
configured remote publish destinations.
### Notarius result handling and reuse
Receive and validate the complete Notarius bundle in Narratio staging. Promote
selected accepted payloads to their flat artifact destinations. Classify the
receipt, warnings, rejection summaries, and diagnostics as Logs. Keep bundle
indexes and other management metadata with the bundle or Narratio State according
to their use.
Revise extraction evidence, catalog hydration, and resume checks to resolve
promoted products independently of the old `artifacts/notarius/` layout. Retain
the identity, provenance, contract, and checksum evidence needed for current
reuse behavior. Historical bundles may remain available for inspection under
their existing retention semantics; relocating them beneath `runs/` must not
silently shorten their lifetime through run cleanup.
Preserve ordinary resume and forced-extraction behavior. Subprocess caches remain
under Notarius's control.
### Inputs
Place prepared previous-session material beneath `inputs/previous/`. Place the
stable reference snapshots Narratio supplies to Notarius beneath
`inputs/references/`, retaining their content identities where needed for reuse.
Update preparation, input resolution, manifest references, and restore mappings.
Keep source configuration ownership and prepared-file validation unchanged.
### Audio acquisition
Simplify acquisition to one prepared session copy under `audio/`. Remove the
separate Narratio S3 audio cache and routine retained spool copies, together with
their obsolete configuration, manifest bookkeeping, and cleanup controls.
Preparation may use temporary download files before installing validated audio.
Preserve reuse of valid prepared audio and update restore to use the same layout.
Recreating a session after its prepared audio is removed requires fetching the
source again. Original recordings remain outside this cleanup change. Removing
spool/cache handling does not authorize new deletion of prepared session audio.
Subprocess caches remain independently configured and managed.
### Logs and execution reports
Provide one consistent latest-attempt view per stage. Include failed attempts;
a skipped stage leaves its previous view intact. Distinguish multiple applications
or artifact invocations within a stage through their filenames. Route Seriatim
reports, Audita reports, Notarius diagnostic records, and Narratio invocation
records through this model. Treat restore execution as its own diagnostic group
under `logs/restore/`, using run-local evidence and a latest-attempt view.
Prevent a new attempt from presenting old diagnostic files as newly generated
ones. At the start of each actual stage attempt, create its run-local log and
debug directories and refresh both latest symlinks. An attempt without debug
output exposes an empty debug directory rather than an older capture. Preserve
subprocess errors and available evidence on failure. Retain existing
captured-stream redaction and limits.
Keep historical files at their run paths; rotating the links does not move files.
Run cleanup must preserve directories targeted by current latest links. Do not
introduce an automatic history age or count limit. Publication and manifest
diagnostic references use actual run paths. Adjust path handling for these
Narratio-managed navigation links without relying on arbitrary symlink traversal.
### Debug controls and placement
Add selective Narratio configuration for Audita and Notarius debug capture using
their supported enabling CLI flags. An enabled Narratio selector requests
capture; an omitted or false selector passes no debug override and preserves the
application's configured/default behavior. Do not add a downstream disable flag.
Supply the actual run-local debug directories, exposed through `debug/polish`
and `debug/extract`. Control the destination when capture can be enabled through
downstream configuration as well, including Audita overlays, using supported
destination controls without changing capture enablement. Audita's debug
work-directory option selects diagnostic storage and belongs to this placement
responsibility.
Route Scriptorium's existing analysis and bounds render-debug products to the
run-local directories exposed through `debug/analyze` and `debug/trim`. Preserve
its existing per-artifact selection capability. Seriatim requires no invented
debug mode; its ordinary reports are
Logs. Subprocess-defined bundle contents and nested run directories are allowed
beneath the selected run-local debug root. Logs and Debug share the same
latest-attempt semantics and cleanup protection.
Keep new full Audita and Notarius debug bundles local in this feature. Preserve
publication of currently declared diagnostics, including existing render-debug
products. Enabling capture does not implicitly add full debug bundles to remote
archives.
Debug controls govern capture when a stage executes. They do not independently
require regeneration of valid substantive outputs. Document the existing force
workflow for obtaining fresh diagnostic evidence when a stage would be reused.
Preserve downstream failure semantics for diagnostic writes and the existing
boundary of Narratio's captured-stream redaction.
### Audita destination support
Request upstream support for independently selecting the transcript output,
ordinary report/log directory, and debug/retained working directory, then adopt
a supporting release. Audita currently ignores `--report-dir` when transcript
output is a file; honoring that destination is the principal missing capability.
Narratio continues to capture subprocess stdout and stderr itself.
Direct ordinary reports, including failure reports, to the actual polish log
directory, the transcript to its staged product destination, and retained debug
material to the actual polish debug directory. This feature uses the supported
upstream controls in a clean cutover without version-based compatibility branches.
### Manifest, publication, restore, and cleanup
Update all consumers of changed paths and file categories together. Preserve
per-speaker and per-artifact completed-work reuse. Keep publication driven by
declared products and diagnostics, with new full debug bundles remaining local.
Restore new-format products and inputs to their new locations.
Define an explicit way to identify supported session state for the clean
cutover. Document regeneration with the new version and manual configuration
updates. Avoid accepting old state through path fallbacks or silently rewriting
old local sessions or remote snapshots.
Preserve existing invalidation and failed-replacement behavior. Review retention
by content role before changing cleanup targets, especially when durable products
or historical bundles move. Run cleanup preserves the current log and debug
link targets; explicit removal of an entire session removes its local views
with it. Subprocess working State and Cache remain outside Narratio's cleanup
responsibility.
### Documentation and verification
Update configuration examples, the operator guide, adapter contracts, workspace
internals, and the architecture policy when implementation lands. Replace the
current architectural assumption that extraction requires a complete immutable
bundle beneath `artifacts/` with the selected product-promotion model. Retire
policy and documentation for Narratio's removed audio cache/spool lifecycle and
document the managed latest-attempt symlinks. Keep per-application cache locations
as deployment recommendations owned by each application's configuration.
Use focused behavioral coverage for flat output placement, collision rejection,
latest-attempt diagnostics on success and failure, debug selection and routing,
unchanged completed-work reuse, failed replacement, and new-format publish/restore.
Cover empty debug views after attempts without capture, preserved views for
skipped stages, cleanup protection of latest targets, and audio acquisition,
reuse, and restore without persistent spool/cache duplicates. Exercise cleanup
against products and separately owned caches. Use temporary filesystems and
subprocess doubles; default tests remain offline and independent
of sibling checkouts or paid providers.