Catalog readability improvements for consumer documentation

docs/roadmap/documentation.md

@@ -1,99 +1,413 @@
-# Documentation Consistency Audit
+# User and Operator Documentation Readability Review
 
-Reviewed revision: `b4477a236aa01e60c17ee353bccaebca7ce919d7`.
+Reviewed revision: `08bb3a20c6a565646d98f68688d268c7bae39573`.
 
-This audit compares current documentation with the implementation, maintained
+Status: proposed revisions only. This catalog identifies opportunities to make
-examples, tests, and architectural decisions. It records proposed corrections;
+existing documentation easier to understand. None of the proposals has been
-it does not change the public contract or implement the corrections.
+applied to its target page. The previous consistency audit remains in Git
+history; its two findings were corrected in `94d1e33`.
 
-## 1. Discovery reference names a removed public constructor
+## Audience and Scope
 
-**Status:** Complete. The discovery introduction now names `SearchPaths`.
+ConfigKit is a library. Developers integrating its public API are its users;
+they need different documentation from contributors maintaining the engine.
+Configuration authors may know YAML without knowing Go reflection or graph
+algorithms. Release maintainers are the audience for its operational runbook.
+ConfigKit has no standalone application CLI or application deployment procedure
+to explain.
 
-**Documentation:** [Configuration discovery](../config.md#configuration-discovery),
+| Material | Primary readers | Treatment in this review |
-line 34, tells consumers to construct a search from literal `Paths`.
+| --- | --- | --- |
+| [Root README](../../README.md) | People evaluating or starting with ConfigKit | Explain outcomes and direct readers to a useful next page. |
+| [Consumer guide](../consumers/README.md) | Application developers adopting or upgrading ConfigKit | Explain tasks and examples before advanced terminology. |
+| [Configuration reference](../config.md) | Configuration authors and API users checking exact rules | Add context while keeping precise contracts available. |
+| [Package documentation](../../doc.go) and public declaration comments | Users browsing the Go API | Explain practical effects alongside symbol names and conditions. |
+| [Release procedure](../release.md) | Maintainers publishing or recovering a release | Clarify operational language while preserving the workflow. |
+| [Release history](../releases) | Users considering an upgrade | Preserve historical records. Future notes can lead with user impact and required action. |
+| [Attribution](../oss/attribution.md), `COPYRIGHT`, and license texts | Users checking rights and notices | Preserve formal notices and license language; no rewrite proposed. |
+| Development guide, internal docs, ADRs, policies, and roadmaps | Repository contributors and decision reviewers | Outside the main rewrite scope; retain links for implementation and rationale. |
 
-**Discrepancy:** The public constructor is now `SearchPaths`, declared in
+The adopted policies should not acquire local wording changes solely for this
-[discovery.go](../../discovery.go), lines 55–56. There is no public `Paths`
+exercise. Accepted ADR decisions and historical release records retain their
-compatibility alias. The private engine retains that name, but consumers cannot
+established treatment under the [documentation policy](../policy/documentation.md).
-use it through the supported public package. The same reference section already
-uses `SearchPaths()` when explaining empty searches, and the
-[migration table](../consumers/README.md#pre-v1-public-api-cleanup) correctly
-records the rename.
 
-**Impact:** A consumer following the introductory instruction can attempt to call
+## Editorial Direction
-an unavailable public API. This is a documentation defect, not a missing API.
 
-**Potential remediation:** Change the introductory constructor name to
+Introduce each topic through the reader's task or the observable result.
-`SearchPaths`. Preserve the historical old-name entry in the migration table and
+Explain unfamiliar terms where they become useful. More words are appropriate
-the private engine name; neither needs changing for this finding.
+when they supply the background needed to understand a rule.
 
-## 2. Four focused-test instructions are truncated in rendered documentation
+Follow introductory explanations with precise reference material. Retain names
+readers will encounter in Go and YAML, and explain them rather than inventing
+new API terminology. Use small illustrations beside the explanation and link
+to maintained files for complete runnable examples. Initially, improve the
+existing pages rather than introducing a new documentation tree.
 
-**Status:** Complete. All four rows now escape their regex pipes consistently
+These are contextual explanations, not global replacement instructions:
-with adjacent rows. Rendered commands retain ordinary regex alternation.
 
-**Documentation:** The [development test map](../development.md#task-to-test-map),
+| Existing term | Useful explanation on first use |
-lines 276, 277, 283, and 290, contains these affected rows:
+| --- | --- |
+| Mapping / sequence / scalar | A YAML group of named values / a list / a single value such as a string or number. |
+| Precedence | Which input takes priority when different layers supply a setting. |
+| Provenance / origin | Where a value came from, including application processing that changed it. |
+| Explicit presence | Whether input supplied a setting, even when its value is `false`, `0`, empty, or null. |
+| Detached record | A returned copy whose mutation cannot change ConfigKit's stored record. Explain borrowed filesystem ownership separately. |
+| Frozen snapshot | The saved configuration state that inspection continues to describe after loading. |
+| Source-local coordinates | Paths within one input document, before its destination is added. |
+| Canonical projection | A consistent representation of typed values used for comparison or inspection; introduce its relevant effect before the internal term. |
+| Dependency-ready order | Process a contribution after its dependencies are ready, using the documented tie-break rule when several are ready. |
 
-- Explicit main-file discovery and absent-input loading.
+## Catalog of Proposed Revisions
-- Import defaults, empty-selection evidence, and dependency-ready ordering.
-- Input and relative-import destinations, mounted sources, and source ownership.
-- Effective configuration diff.
 
-**Discrepancy:** These rows put unescaped regex alternation pipes inside inline
+The identifiers below name proposals, not implementation stages. High priority
-code in Markdown table cells. GitHub-flavored Markdown treats those pipes as cell
+means a proposal helps readers understand a common or foundational task. Medium
-separators even inside backticks. The table has three columns, so the remainder of
+priority addresses a specialized task or reference detail.
-each instruction falls outside the displayed command cell. For example, the
-discovery command renders as the incomplete text
-`` `go test ./internal/engine -run 'Discovery `` rather than the complete command
-and its two companion commands.
 
-The intended Go regexes are valid: `|` separates alternatives. Rendering the
+### D01 — Introduce ConfigKit through its useful outcomes
-current guide with the already module-selected Goldmark v2.1.6 GFM parser and
-renderer reproduced all four truncated cells. Neighboring older rows escape
-their pipes and render complete commands. Existing tests referenced by these
-instructions run successfully in the complete suite; their implementation is
-not the problem.
 
-**Impact:** Readers of the rendered contributor guide lose the complete test
+**Target:** [README introduction](../../README.md). **Priority:** High.
-selection instructions and can copy an unterminated shell command. The existing
-documentation-link check does not validate rendered table cells.
 
-**Potential remediation:** Escape the table pipes consistently with adjacent
+The feature inventory precedes the minimal example and introduces "snapshotted
-rows, or move the affected commands into fenced blocks outside the table. Inspect
+mappings," "detached metadata," "audited defaulting," and "source/derivation
-the rendered result and execute the resulting shell commands with ordinary regex
+history" in quick succession.
-alternation. Markdown escaping should not introduce literal escaped pipes into
-the command users copy. No new release gate or production change is needed.
 
-## Review Coverage and Validation
+Lead with what the application receives and what its author supplies. Suggested
+direction: "ConfigKit loads YAML into your application's Go configuration type,
+combines the inputs you choose, and lets you inspect where the final values came
+from." Bring the existing minimal example forward. Follow it with short
+capability summaries and links, introducing optional features when readers
+select them. Keep platform and licensing navigation easy to find. Detailed
+scheduling and ADR links can follow the consumer-oriented navigation.
 
-Reviewed the root README and package/API documentation, the configuration
+### D02 — Make the consumer guide a route through common tasks
-reference, consumer guide and executable examples, development instructions,
-internal component/import documentation, and release procedure against their
-corresponding implementation and tests. Checked module identity, declared Go and
-ReleaseKit versions, release-profile wiring, and licensing/attribution navigation.
-No additional actionable discrepancy was identified in those areas. This was not
-an upstream licensing review or an attempted release.
 
-Historical release notes, dated extraction records, superseded ADR decisions,
+**Target:** [Consumer introduction and entry-point selection](../consumers/README.md#choosing-a-loading-entry-point),
-explicitly proposed ADRs, migration tables, and deferred roadmap candidates were
+the example-shape table, and Happy path. **Priority:** High.
-treated according to their stated status. Their historical terminology and
-proposed behavior are not findings merely because current behavior differs.
 
-Validation used Go 1.26.10 with local toolchain selection and dependency downloads
+The guide begins with a long responsibility inventory and examples named after
-disabled:
+other applications. Its happy-path snippet immediately presents imports,
+overlays, overrides, and a predeclared rules value.
 
-```sh
+Offer a short task index: load one file, supply defaults and validation, add
-GOTOOLCHAIN=local GOPROXY=off go test ./... -count=1
+bundled lower-priority settings, select an overlay, and inspect a result. Point
-GOTOOLCHAIN=local GOPROXY=off go test . -run 'TestLocalDocumentationLinks|TestDependencyImportBoundary|TestPublicFacadeBoundary|^Example' -count=1
+the simplest case to the README example, then introduce additional options as
-```
+needed. Describe examples by their demonstrated task before their original
+application names. Move extraction background and test-owner inventories below
+usage guidance. Preserve links to maintained examples.
 
-Both passed, including maintained consumer examples. An external disposable GFM
+### D03 — Give the reference useful starting points
-rendering probe substantiated finding 2. No production code, existing tests,
-current documentation contracts, release tags, or consumer dependencies were
-changed during the audit.
 
-Remediation validation: all 13 commands in the four corrected rows render in
+**Target:** [Configuration reference introduction](../config.md). **Priority:** High.
-full, with ordinary regex alternation. All 11 distinct rendered commands passed,
+
-as did the local documentation-link check and whitespace review.
+The opening says the document "owns" contracts and describes private parser
+boundaries. The page mixes basic input selection with advanced lifecycle and
+inspection details across a long reference.
+
+Introduce it as the reference for accepted YAML, input combination, and loading
+behavior. Add a compact topic index with routes for configuration authors and
+Go API users. Introduce mappings, lists, layers, and configuration paths with
+small illustrations. Separate common behavior, advanced interactions, and limits
+through subheadings within the existing document. Keep detailed rules available
+after their introductions; a separate general-purpose glossary is unnecessary
+initially.
+
+### D04 — Explain source selection and ownership in practical terms
+
+**Target:** [Sources](../config.md#sources),
+[single-source loading](../config.md#single-source-loading), and
+[resource lifetime](../config.md#completion-and-resource-lifetime).
+**Priority:** High.
+
+"Opaque source," "consumption," "claimed once," "defensive source list," and
+"publication copy" put implementation vocabulary ahead of caller actions.
+
+Introduce sources as inputs passed to `Load`. Explain which are copied
+immediately, which are read again on each load, and why a `Reader` source is
+single-use. Keep the constructor table and explain who closes each resource.
+Introduce the result before explaining copying mechanics. Suggested wording:
+"You can change `Result.Config` after loading. Inspection methods continue to
+describe the configuration as it was when the load finished." Preserve the
+separate rule that borrowed filesystems remain under caller control and later
+resource reads observe those filesystems.
+
+### D05 — Teach combination rules with visible input and output
+
+**Target:** [README philosophy](../../README.md#design-philosophy),
+[layered loading](../config.md#layered-loading),
+[strict composition](../config.md#strict-document-composition), and
+[defaults](../config.md#defaults-and-typed-decoding). **Priority:** High.
+
+Readers encounter "tier," "strict peers," "recursive precedence," and "typed
+baseline" before seeing a combined configuration. Priority order and processing
+order are also easy to confuse.
+
+Use a small mapping example to show independent same-tier contributions and an
+intentional higher-layer change. Explain same-tier conflicts. Show a priority
+diagram separately from the processing timeline, explaining why defaults have
+low priority while being applied during decoding. Keep the README summary brief
+and link to full reference rules. Add a defaults example distinguishing omission,
+explicit false/zero, an empty list, and null where the Go field permits it.
+Keep collection exceptions visible; avoid a blanket "last file wins" explanation.
+
+### D06 — Describe discovery as an ordered search
+
+**Target:** [Configuration discovery](../config.md#configuration-discovery) and
+[consumer discovery guidance](../consumers/README.md#explicit-main-file-discovery).
+**Priority:** High.
+
+"Directory-first Cartesian product," "ordered concatenation," and "optional
+exhaustion" require translation into actual attempted filenames.
+
+Show two directories and two filenames as an ordered list of candidates.
+Explain that ConfigKit finishes the filenames for one directory before moving
+to the next. Explain no-match behavior, required searches, and selected-file
+failure. Keep the terminal selected-file error rule prominent so readers
+understand why a bad selected file does not cause another search. Follow with
+the existing preset table and specialized filesystem-context rules.
+
+### D07 — Introduce imports through the three selection tasks
+
+**Target:** [Explicit imports](../config.md#explicit-imports) and
+[import defaults guidance](../consumers/README.md#import-defaults-and-selection-inspection).
+**Priority:** High.
+
+The opening combines enablement, structural controls, selector shapes,
+requiredness, destinations, removed syntax, and diagnostic codes. Later passages
+refer to "namespace-path lexical order" and "scoped assembly order."
+
+Start with including a named file, searching a directory, and matching a path
+pattern. Use small YAML illustrations and link to complete examples. Explain
+shared options in a selector-applicability table. Separate optional missing
+inputs from unreadable or invalid inputs. Explain that required path lists check
+each named target, while required directory/glob searches need a selected file;
+retain the empty `path: []` case. Explain observed ordering with a short dependency
+example before its exact reference rules. Move legacy migration notes below the
+first explanation of current syntax.
+
+### D08 — Separate configuration placement from filesystem paths
+
+**Target:** [Input destinations](../config.md#input-destinations),
+[resource contexts](../config.md#source-resource-contexts), and
+[mounted collection guidance](../consumers/README.md#input-destinations-and-mounted-collections).
+**Priority:** High.
+
+"Effective destination," "projected through that item," "nearest resolved
+enclosing collection container," and "passive resource context" are difficult
+without a picture of the configuration and the files on disk.
+
+Show a tiny file tree beside a resulting YAML shape. Explain that a destination
+chooses where settings appear, while an import filename resolves from its
+declaring file. Walk through opening one resource: follow its value's source ID
+to `SourceContext`, join `BaseDir`, and close the file the application opens.
+Introduce fallback destination evidence after exact placement. Preserve
+composite-origin ambiguity, caller-owned resource lifetimes, and the distinction
+between saved configuration and later resource I/O.
+
+### D09 — Explain collection identity and operations before policy lifetime
+
+**Target:** [Collection rules](../config.md#strict-document-composition) and
+[identified sequence guidance](../consumers/README.md#identified-prompt-sequences).
+**Priority:** High.
+
+Selectors, normalized field sets, type-tagged tuples, registrations, and retained
+policy appear before the operation table. Readers who want to combine a list of
+named records need substantial background first.
+
+Start with a record list and an `id` field. Show before/after results for replace,
+append, and upsert, including an upsert whose omitted lower fields disappear
+during source assembly. Explain duplicate rejection within a strict tier. Next
+introduce YAML declarations and Go registrations, including their different
+final-validation guarantees. Label compound identity, exact numeric equality,
+wildcard selectors, pending declarations, and record-owned policy as advanced
+reference material. Preserve the existing operation and presence tables.
+
+### D10 — Introduce inheritance with a parent and child
+
+**Target:** [Sibling inheritance](../config.md#sibling-inheritance) and
+[consumer inheritance guidance](../consumers/README.md#named-sibling-inheritance).
+**Priority:** Medium.
+
+The opening quickly moves from named siblings to identified records, root
+restrictions, cancellation, and dependency scheduling.
+
+Begin with a small `base`/`custom` mapping example and show the resulting child.
+Explain "sibling" as another entry in the same enclosing mapping. Explain that
+inheritance uses the settings remaining after source combination. Add
+cancellation and identified-record examples next, followed by exact ordering.
+Retain destination-local nested resolution, identity-field ownership,
+cycle/missing-parent errors, and ordinary validation of parent records.
+
+### D11 — Explain version checking as application-defined acceptance
+
+**Target:** [Application versions](../config.md#application-schema-versions),
+[shared checks](../config.md#shared-source-version-checks), and
+[consumer version guidance](../consumers/README.md#shared-source-version-checks).
+**Priority:** Medium.
+
+"Accepted non-main declarations are consumed from their owned payload" and
+"destination-prefixed application path" combine several concepts in one sentence.
+
+Explain first that the application chooses the version field and supported
+identifiers. Add a behavior table for the three modes, distinguishing missing
+main headers, missing other headers, explicit null, and `NoSource`. Use a mounted
+document example to separate checking a header inside a file from protecting its
+assembled path. Explain that accepted non-main headers are removed before
+combination. Preserve the distinctions between observing a declaration, running
+a checker, and acceptance, including disclosure limits on `VersionEvidence`.
+
+### D12 — Explain callbacks through the application's work
+
+**Target:** [Recorded resolution](../config.md#recorded-application-resolution),
+[semantic validation](../config.md#semantic-validation), and consumer task guidance.
+**Priority:** High.
+
+"Audited net resolution operations," "helper-assigned values," "covering record,"
+and "post-decode entry state" obscure when to use each callback.
+
+Introduce fixed defaults, deriving or normalizing values in `Resolve`, and
+checking rules in `Validate` as separate tasks. Compare `Default`, `Normalize`,
+and the two `Record*` methods, identifying which assign values. Explain that
+`EntryInfo` describes callback-entry state even after the resolver changes
+values. Use a small derived-value example to explain recording and dependencies.
+State callback lifetime and read-only validation rules directly. Retain panic,
+error-reporting, and helper-writeback behavior in the detailed reference.
+
+### D13 — Separate secret references, hiding output, and file bootstrap
+
+**Target:** [Typed references](../config.md#typed-secret-references),
+[classification/rendering](../config.md#secret-classification-and-effective-rendering),
+and [file bootstrap](../config.md#explicit-secrets-directory-bootstrap).
+**Priority:** High.
+
+"Credential-bearing fields," "sensitivity closure," "preflight/apply," and
+"existing empty winner" demand familiarity with several different mechanisms.
+
+Explain the YAML environment-variable name, the credential read during loading,
+and the separate choice to hide a configuration field in inspection. Make clear
+that reference names are visible unless separately classified. Explain the
+optional helper as reading files into environment variables before a later load,
+with a filename/name/value illustration using dummy data. Explain
+`skipped_existing` with the already-set-but-empty case, then show how to read a
+partial report. Preserve partial writes, lack of rollback, lookup timing,
+application-owned safe messages, and visibility limits. Keep exact byte/name
+limits in their reference paragraphs or tables.
+
+### D14 — Explain inspection through the information readers need
+
+**Target:** [Metadata](../config.md#layered-metadata),
+[explanations](../config.md#historical-value-explanation),
+[diffs](../config.md#effective-configuration-diffs), and their consumer examples.
+**Priority:** High.
+
+"Canonical typed projection," "decoding reconciliation," and "cross-protected
+context" are demanding introductions to inspecting a setting.
+
+Organize the introduction around the final value, its source, its earlier
+definitions, and changes between two loads. Walk through a setting with a
+default, supplied value, and later override. Distinguish metadata history from
+value explanations. Explain `Present`, `Redacted`, and `HiddenChanged` with a
+small table and hidden-change example. Keep source-versus-typed representation
+details, atomic list comparisons, combined sensitivity, and unchanged-reference
+credential rotation. Clarify which outputs contain values and which contain
+attribution only.
+
+### D15 — Lead YAML and Go type rules with authoring examples
+
+**Target:** [YAML documents](../config.md#yaml-documents),
+[scalars](../config.md#scalars), [schema binding](../config.md#application-schema-binding),
+[decoding](../config.md#defaults-and-typed-decoding),
+[projection/ownership](../config.md#typed-projection-and-ownership), and
+[safe failures](../config.md#safe-failures). **Priority:** Medium.
+
+"Private reflection descriptor graph," "paired text scalars," and "recursive
+acyclic objects" precede common authoring needs. Safe failures explains
+classifications before showing how to use a report.
+
+Start with a YAML/Go field pairing and everyday examples: quote `yes` to use it
+as text, write durations as strings, and use supported lowercase booleans.
+Retain the scalar/defaults tables. Explain nil versus empty containers, ignored
+fields, and `omitempty` through visible results. Put codec method sets and
+recursive-type rules in advanced reference material. Explain diagnostic code,
+configuration path, and source location with a small safe report and a link to
+maintained error handling. Retain exact limits and error behavior. Move purely
+internal caching or tree-storage explanations to existing internal docs where
+useful, preserving all observable ownership guarantees.
+
+### D16 — Make Go API comments understandable on their own
+
+**Target:** [doc.go](../../doc.go), public declarations in
+[lifecycle.go](../../lifecycle.go), [path.go](../../path.go),
+[records.go](../../records.go), [inspection.go](../../inspection.go), and
+[diff.go](../../diff.go). **Priority:** Medium.
+
+The package comment is chiefly a feature inventory. `Resolution` introduces
+"entry-state input provenance"; `Difference` introduces a "canonical changed
+region"; several result comments rely on "detached" without explaining its effect.
+
+Give major types an action-oriented first sentence followed by their precise
+contracts. Illustrate configuration paths with a key/index example, and explain
+returned-copy and saved-state behavior where relevant. Keep symbol-specific
+errors, nil/zero behavior, ownership, and lifetime rules beside declarations.
+Link to shared YAML semantics and maintained examples rather than repeating the
+full configuration reference. Preserve public names and Go doc conventions.
+
+### D17 — Turn migration prose into recognizable consumer actions
+
+**Target:** [Migration changes](../consumers/README.md#intentional-migration-changes),
+[module-path migration](../consumers/README.md#vanity-module-path-migration), and
+[API cleanup](../consumers/README.md#pre-v1-public-api-cleanup).
+**Priority:** Medium.
+
+The old/new symbol table is useful, but "serialized diagnostic/provenance
+expectations" and "migration evidence" assume repository history and advanced
+Go package knowledge.
+
+Lead with actions: update requirements and imports together, replace listed old
+symbols, and review any stored or compared layer/code strings. Explain distinct
+Go type identity through another dependency passing ConfigKit values to the
+application. Retain the old/new table, historical-tag limitations, and the
+difference between API and YAML changes. Keep downstream migration-status caveats
+in a short background note. Do not choose a release version or claim that another
+application has migrated.
+
+### D18 — Make the release runbook easier to follow and recover from
+
+**Target:** [Release procedure](../release.md), especially Go selection,
+publication status, and recovery. **Priority:** Medium.
+
+"Module-selected native executable," "consumer minimum," "bounded authoritative
+reconciliation," and "peeled commit" require substantial release-tool knowledge.
+
+Explain a candidate as the committed revision intended for release, the consumer
+minimum as the oldest supported Go version, and the selected tool as the
+ReleaseKit version recorded in `go.mod`. Use the existing Go patch example to
+distinguish validation toolchain from consumer minimum. Keep command blocks
+intact unless a separately verified wording change needs an accompanying
+adjustment. Put exit statuses and next actions in a compact table. Explain why
+the tag-object ID matters before asking the operator to compare it. Keep
+recovery's exact URL, candidate, and tag identity conditions together.
+
+Preserve pushing the candidate before publishing the tag, publication
+authorization, immutable published tags, a single publication command,
+and optional consumer verification. Keep CI waiting and macOS validation outside
+the release gates. Plain language must not turn uncertain publication into
+permission to retry, delete a published tag, or declare success.
+
+## Priorities and Review Boundaries
+
+The highest-impact starting points are the README and consumer entry routes
+(D01–D03), combination/presence explanations (D05), and secret/inspection guidance
+(D13–D14). The other proposals develop a selected task or reference topic. These
+priorities are editorial recommendations, not additional release gates.
+
+Later rewrites should preserve existing contracts, diagnostic identifiers,
+defaults, limits, security boundaries, and exceptions. Keep precise material
+available after its plain-language introduction. A simpler explanation must not
+imply universal last-value replacement, universal null support, transactional
+loading, refreshed inspection after caller mutation, or stronger filesystem
+isolation.
+
+This pass changes only this catalog. No user-facing page, Go comment, example,
+policy, command, test, or implementation has been revised. Later implementation
+can validate changed examples and commands at their existing owners and inspect
+rendered tables and navigation under the documentation policy.