From f6f41ee6aa9346bc424cee723b06c770d2a7aec8 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 6 Nov 2026 03:38:12 +0000 Subject: [PATCH] Fold configuration discovery and diagnostics into existing commands --- docs/roadmap/future.md | 5 +- docs/roadmap/promptkit-v0.19.0.md | 140 +++++++++++++++++++----------- 2 files changed, 92 insertions(+), 53 deletions(-) diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index baa660c..ef0c1f5 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -55,7 +55,10 @@ retaining those proposals does not change current comparison behavior. The [PromptKit v0.19.0 roadmap](promptkit-v0.19.0.md#shared-configuration-capabilities) now includes the shared operator tasks previously retained here: metadata/path discovery and explicitness, failed-load source inventory, detailed comparison, -and structured diagnostics. It also includes focused effective display, +and structured diagnostics. Path discovery is a mode of `config explain`, and +structured diagnostics belong to `config validate`; the suite retains six +subcommands. Their exact mode flags and report shapes remain open in the roadmap. +It also includes focused effective display, source-ID selection, metadata/diagnostic filters, independent comparison selectors, and assertion mode. The original four shared proposals had ConfigKit adoption identifiers OD-3 through OD-6; the new roadmap has its own decision numbering. Implementation diff --git a/docs/roadmap/promptkit-v0.19.0.md b/docs/roadmap/promptkit-v0.19.0.md index c692efe..73f742a 100644 --- a/docs/roadmap/promptkit-v0.19.0.md +++ b/docs/roadmap/promptkit-v0.19.0.md @@ -97,14 +97,21 @@ the decisions at the end of this document. | Inspect what a concrete prompt request prepares | `promptkit show request ID --profile … --input … --var …` | `ckprompt`: selected IDs, effective settings, rendered/appended messages, output contract and schema, session, hashes, and timing from `PreparedRun`. | | Reproduce target/provider settings without editing configuration | Applicable existing smoke override flags on validation and preview; `--extra-params JSON` on these and smoke operations | `ckprompt`: native request forwarding; preview and smoke reports show returned effective settings. | | Resolve unqualified resource selections in chosen namespaces | Independent prompt/profile/backend namespace flags | `ckprompt`: delegate to native namespace views, retaining the distinction from catalog filters. | -| Find configuration paths and inspect their explicitness | `config paths --scope promptkit`, with historical selection; metadata alongside path explanation | `ckconfig`, wired through `ckprompt.ConfigScope`: observed paths, effective presence, explicitness/null/origin evidence, and available history. | +| Find configuration paths and inspect their explicitness | `config explain --list --scope promptkit`, with historical selection; metadata alongside path explanation | `ckconfig`, wired through `ckprompt.ConfigScope`: observed paths, effective presence, explicitness/null/origin evidence, and available history. | | Understand both sides of configuration changes | `config diff --scope promptkit --details` | `ckconfig`: native paths, left/right snapshots, presence/redaction, and hidden-change evidence. | -| Investigate a failed configuration load | `config sources --scope promptkit` on failure; a structured diagnostics operation | `ckconfig`, with both native adapters: acquired source inventory and native codes, severities, paths, and locations while retaining the load failure. | +| Investigate a failed configuration load | `config sources --scope promptkit` on failure; structured diagnostics within `config validate` | `ckconfig`, with both native adapters: acquired source inventory and native codes, severities, paths, and locations while retaining the load failure. | | Inspect one effective value or subtree | `config show --scope app --path /server` | `ckconfig`: the selected native effective snapshot, retaining redaction and absence/null distinctions. | | Follow a source reference | `config sources --source-id ID` | `ckconfig`: the matching acquired source record, including evidence available after failure. | | Compare different application selections | For example, `config diff --campaign spring --against-campaign autumn` | `ckconfig`: independently prepared typed inputs passed to the existing native comparison provider. | | Detect configuration differences in scripts | `config diff --check` | `ckconfig`: a deliberate difference outcome based on native records, alongside the selected comparison report. | -| Narrow an inspection to relevant evidence | Subtree/explicitness filters on `config paths`; severity/code filters on `config diagnostics` | `ckconfig`: filtered native evidence with source references retained. | +| Narrow an inspection to relevant evidence | Subtree/explicitness filters on explanation discovery; severity/code filters on validation diagnostics | `ckconfig`: filtered native evidence with source references retained. | + +The configuration suite retains six subcommands: `scopes`, `validate`, `show`, +`sources`, `explain`, and `diff`. Path discovery belongs within `config explain`; +structured diagnostics belong within `config validate`. This command placement +is agreed. Mode flags such as `--list` and `--diagnostics`, report shapes, and +filter spellings remain proposals under OD-4, OD-6, and OD-8. No separate path +inventory or diagnostics subcommand is planned. Transient configuration overrides are an open scope decision under OD-11, with a recommendation to include them as an optional loader capability. Illustrative @@ -199,14 +206,17 @@ they have a defined use; define that applicability table during implementation. Extend `ckconfig` through optional capabilities so existing custom providers can keep their currently declared operations. Both `FromConfigKit` and `ckprompt.ConfigScope` should expose available native evidence. Do not require -all providers to implement every new operation. +all providers to implement every new inspection mode. Providers lacking metadata +or structured diagnostics retain their existing explanation/validation behavior; +mode availability and help must reflect the capabilities supplied. For metadata, use `Metadata.HistoricalPaths`, `Lookup`, `IsExplicit`, and `History`. Enumeration covers observed paths, not all possible schema fields. Do not discard history because effective lookup fails. Existing explanation already reports current/historical values; add the missing discovery and -explicitness instead of reproducing upstream provenance logic. Preserve native -list-index paths; do not silently reinterpret them as resource IDs. +explicitness within explanation discovery instead of reproducing upstream +provenance logic. Preserve native list-index paths; do not silently reinterpret +them as resource IDs. For detailed comparison, project native `Difference` records. Keep snapshots and their presence/redaction fields together, including `HiddenChanged` when @@ -225,7 +235,11 @@ successful report delivery must not convert failure to success. Preserve render/write failures separately through the current outcome machinery. Cover cancellation with partial evidence explicitly without promising interruption of an arbitrary writer or rollback of already-delivered bytes. OD-6 selects the -structured diagnostics command and its interaction with ordinary diagnostics. +validation diagnostic mode and its interaction with ordinary diagnostics. +Structured records describe that validation invocation, including a failed +load; they are not a retrieval interface for an earlier command. Preserve the +current aggregate validation stopping behavior unless deliberately revised; +structured reporting alone does not promise exhaustive findings across scopes. ### Additional ConfigKit operator workflows @@ -286,9 +300,10 @@ errors remain errors even if an override would replace the affected value. public `ckprompt` execution-flag collections on existing core bindings and reuse them in the standard commands. Reuse the standard suite's lifecycle and preserve existing application adapter seams. -4. **Implement shared configuration inspection.** Add metadata discovery and - explicitness, detailed diff presentation, source evidence on failure, and - structured diagnostics in `ckconfig`; wire native PromptKit facets through +4. **Implement shared configuration inspection.** Add discovery and metadata + within `config explain`, structured diagnostics within `config validate`, + detailed diff presentation, and source evidence on failure in `ckconfig`. + Retain the six-command suite and wire native PromptKit facets through `ConfigScope`. Add focused effective display, source selection, and bounded metadata/diagnostic filters using those evidence capabilities. Extend typed comparison selection and add the selected assertion outcome. If OD-11 is @@ -333,8 +348,12 @@ Default tests remain offline and independent of real credentials. the collection. - [ ] Namespace defaults, explicit clearing, qualified selections, and catalog filters are forwarded according to the selected design. -- [ ] Metadata discovery distinguishes effective and historical paths, retains - available history on absent lookup, and projects native field/change records. +- [ ] The full configuration suite retains six subcommands. Explanation + discovery and validation diagnostics are modes of existing commands, with + capability-aware help and selected mode/flag combination checks. +- [ ] Metadata discovery through `config explain` distinguishes effective and + historical paths, retains available history on absent lookup, and projects + native field/change records. - [ ] Detailed diff retains both snapshots and presence/redaction/hidden-change evidence, including empty comparisons and sequence-level changes. - [ ] Focused display preserves native snapshot content and distinguishes @@ -352,7 +371,8 @@ Default tests remain offline and independent of real credentials. overrides reach opted-in loaders, with typed values, safe flag labels, selected comparison-side behavior, native errors, and returned provenance preserved. No tests independently re-establish ConfigKit override semantics. -- [ ] Source and diagnostic evidence can be reported with an acquisition error; +- [ ] `config sources` and the structured mode of `config validate` report + available evidence with an acquisition error; the original failure, unavailable-versus-empty evidence, cancellation, and report-delivery failures remain distinguishable. - [ ] Existing providers remain usable; help, version, and capability discovery @@ -462,17 +482,23 @@ must account for the native API replacing all three defaults at once. ### OD-4: Configuration path and metadata presentation -**Recommendation:** add `config paths` with effective paths by default and an -explicit historical selection, and expose optional metadata alongside existing -path explanation. Use the current single-scope selection model. This makes -paths discoverable and uses an established command for their meaning/history. -Define availability and unavailable metadata explicitly for older providers. +**Agreed placement:** path discovery is part of `config explain`; no separate +path or metadata command will be added in this roadmap. -**Alternative:** a separate `config metadata --path …` operation plus historical -enumeration. This keeps the existing explanation report unchanged and gives -metadata-only providers an independent operation, but splits related operator -information across commands. Weigh that compatibility benefit against the -extra command and report contract before selecting additive explanation fields. +**Recommendation:** add an explicit discovery mode such as +`config explain --list`, with effective paths by default, optional historical +selection, and subtree/explicitness filters. Keep `config explain --path POINTER` for detailed +explanation, including optional field metadata. This retains the current +missing-path usage error outside discovery mode and makes the operator's intent +clear. Use the current single-scope model. Define mode combinations, formats, +and unavailable metadata handling before implementation; exact flag spellings +remain open. + +**Alternative:** make omission of `--path` select discovery automatically, while +supplying it selects detailed explanation. This reduces flags and gives bare +explanation a useful behavior, but deliberately changes today's missing-path +error and can make an accidentally omitted path harder to notice. Both designs +keep discovery and detailed history together in one command. ### OD-5: Detailed comparison selection and shape @@ -489,22 +515,31 @@ justify that disruption. ### OD-6: Failure evidence and structured diagnostic delivery -**Recommendation:** extend `config sources` to report acquired inventory on -failure and add `config diagnostics --format yaml` for native structured records -and their source legend. Build this as optional `ckconfig` evidence facets, -including evidence returned with an acquisition error. Keep ordinary human -messages on the diagnostic stream and structured results on the result stream, -using existing terminal failure/outcome handling. This is a bounded operator -workflow with no demonstrated need for a generic core diagnostics schema. +**Agreed placement:** structured configuration diagnostics are part of +`config validate`; there will be no separate diagnostics command. Source +inventory on failure remains part of `config sources`. -**Alternative:** add structured diagnostic output to every configuration -operation through a common format option or report envelope. It avoids a -second invocation when diagnosing another command, but complicates existing -native YAML/report shapes and result-versus-diagnostic stream contracts. A -broader core representation is viable if concrete cross-family needs justify -it; it is more work than exposing ConfigKit records alone. Specify warning -presentation, evidence availability, and partial-result cancellation behavior -for the selected design rather than silently changing all operations. +**Recommendation:** add an explicit validation diagnostic mode, for example +`config validate --diagnostics --format yaml`, reporting native records and a +source legend alongside the validation result. Keep the existing validation +report when that mode is omitted. Support severity/code filters in the mode +without changing the underlying outcome. This preserves existing consumers and +lets the same invocation both validate and explain its findings, including +available evidence on load failure. + +Build optional `ckconfig` evidence facets that can accompany an acquisition +error. Structured results use the result stream, ordinary messages use the +diagnostic stream, and terminal failure/outcome handling remains in place. +Resolve warning duplication, unavailable-versus-empty evidence, report shape, +and partial-result cancellation behavior during design. Successful delivery of +findings must not replace a failed load with a successful validation result. + +**Alternative:** enrich `config validate --format yaml` directly with diagnostic +records and source evidence, without a new mode flag. This gives scripts richer +information by default but changes the existing YAML report and failure-output +contract, requiring deliberate compatibility treatment. Either choice stays +within validation and uses current core reporting; a generic cross-family core +diagnostics schema is not needed for this scope. ### OD-7: Module release and interim workspace coordination @@ -527,21 +562,23 @@ the declared released dependency. **Recommendation:** extend existing commands: `config show --path POINTER` and `config sources --source-id ID`; add subtree/explicitness filters to -`config paths` and severity/code filters to `config diagnostics`. Keep full reports as -the default. Use the existing explanation capability for selected effective -YAML, and recommend an operational failure for an absent effective path or +explanation discovery and severity/code filters to validation diagnostics. Keep +unfiltered reports as the default within those modes. Use the existing +explanation capability for selected effective YAML, and recommend an operational +failure for an absent effective path or unknown source ID, consistent with unknown-path explanation. Recommend starting path-selected display with one scope; whole-document `--scope all` remains available. Specify presence-aware explicitness filtering so historical-only entries are not labeled implicitly defined. These choices reuse existing operator vocabulary and native evidence with little additional acquisition work. -**Alternative:** introduce a `config get --path POINTER` command with a snapshot -envelope containing `present`, `redacted`, and `yaml`, and return absence as -report data. This is convenient for scripts and preserves unknown/null/absence -distinctions without relying on exit status, but adds another command/report -contract. Aggregate path-selected reports are also viable if scope-labeled -per-scope absence and delivery semantics are designed. Filters can be deferred +**Alternative:** offer a structured mode of `config show --path POINTER` with +a snapshot envelope containing `present`, `redacted`, and `yaml`, and return +absence as report data. This is convenient for scripts and preserves +unknown/null/absence distinctions without relying on exit status, but adds a +report format choice alongside native effective YAML. Aggregate path-selected +reports are also viable if scope-labeled per-scope absence and delivery semantics +are designed. Filters can be deferred in favor of downstream YAML tools, reducing CLI surface but making routine operator inspection more cumbersome. @@ -573,11 +610,10 @@ reporting outcome rules, including preservation of a nonzero domain outcome when presentation also fails; callers can inspect the separate presentation evidence. Document that interaction explicitly. -**Alternative:** add a separate `config compare` assertion command, leaving -`diff` exclusively informational. This makes the intent visible in the command -path but duplicates selection/report options. Another viable choice is status -`1` for both differences and operational failures, which fits simple shell -checks but requires additional evidence to distinguish the two conditions. +**Alternative:** use status `1` for both differences and operational failures +in the opt-in assertion mode. This fits simple shell checks and avoids a new +difference status, but requires additional evidence to distinguish detected +changes from failed comparison. Both approaches remain modes of `config diff`. ### OD-11: Transient configuration overrides