From ed6060f344efd2841aa8a23b9535e660dec5d232 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 16 Nov 2026 17:36:31 +0000 Subject: [PATCH] Plan invalid configuration outcome handling --- .../roadmap/invalid-configuration-outcomes.md | 237 ++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 docs/roadmap/invalid-configuration-outcomes.md diff --git a/docs/roadmap/invalid-configuration-outcomes.md b/docs/roadmap/invalid-configuration-outcomes.md new file mode 100644 index 0000000..503f01c --- /dev/null +++ b/docs/roadmap/invalid-configuration-outcomes.md @@ -0,0 +1,237 @@ +# Invalid configuration outcomes + +Status — proposed implementation roadmap. The user has accepted the targeted +integration behavior. The decisions at the end identify remaining policy and +API choices for the implementation plan. + +## Purpose + +Make standard ckconfig and ckprompt commands return usage status `2` for +identifiable invalid configuration input, while returning operational status `1` +for cancellation and other operational failures. Preserve actionable upstream +diagnostics and supported typed error identity through the integration boundary. + +CommandKit owns this CLI classification. ConfigKit and PromptKit own loading, +validation, diagnostic meaning, and their public error contracts. Application +commands own their domain outcomes. These responsibilities follow the +[architecture policy](../policy/architecture.md#ownership-boundaries). + +## Current baseline + +The reviewed baseline is CommandKit commit +`1e92a87706e44684ee34395245e5b633474677e0`, following the publication of +ckconfig v0.7.0 and ckprompt v0.7.0. Both integration modules select core +v0.6.0 and ConfigKit v1.2.0. ckprompt selects ckconfig v0.7.0 and PromptKit +v0.21.0. Their manifests own the dependency selections. + +The consumer report reproduced malformed operator YAML in Audita, including +its automatic `prompts.yml` companion and an explicitly selected application +configuration. Delegated inspection returned `yaml.invalid` with status `1`, +while Audita's processing command returned the same diagnostic with status `2`. +The report's reproduction is retained in Audita's +[upstream handoff](https://forge.maximumdirect.net/eric/audita/src/commit/a355f05a273d6ac9d577eab8d30caa60cb74eee8/docs/roadmap/implementation.md). + +The review confirmed the behavior through Audita's existing regression tests and +a temporary public-API reproduction against the aligned CommandKit workspace. +The latter covered catalog listing and all five configuration operations in both +the application and PromptKit scopes, including wrapped ConfigKit loading errors +and already-canceled contexts. + +- [FromConfigKit](../../integrations/ckconfig/configkit_adapter.go) and + [the PromptKit adapters](../../integrations/ckprompt/adapters.go) preserve the + ConfigKit loading error and adapt its diagnostic evidence. +- [configFailure](../../integrations/ckconfig/operations.go) and + [definitionFailure](../../integrations/ckprompt/command.go) construct operational + outcomes for the affected failures. Configuration comparison and validation + also reach the former helper. +- ConfigKit's public `LoadError` represents several kinds of loading failure, + including malformed YAML, acquisition errors, and schema or callback problems. + Its public diagnostic codes provide the classification evidence. Its error + contract intentionally exposes safe diagnostics and preserves its own typed + identity across CommandKit adaptation. +- `config sources` can report discovered source evidence after failed loading, + and `config validate` can report failed validation. Cancellation observed + before acquisition leaves stdout empty in the reviewed command paths. + +The current [CLI contract](../cli.md#outcome-status) assigns loading failures +status `1`, with more specific statements in the configuration comparison, +requirements, validation, and smoke sections. This roadmap proposes a deliberate +change to that published status behavior. + +## Targeted end state + +| Evidence available to the integration | Selected outcome | +| --- | --- | +| ConfigKit error diagnostics that identify invalid configuration input under the selected mapping | Usage, status `2` | +| Caller cancellation or deadline observed at an existing operation checkpoint, or present in the returned error chain | Operational, status `1` | +| I/O, schema or wiring defects, callback execution failures, and other recognized operational failures | Operational, status `1` | +| An unrecognized failure, including a loading error with insufficient classification evidence | Operational, status `1` | +| Successful acquisition with warnings | Follow the command's successful-result behavior | + +Classification uses upstream structured evidence. Diagnostic text, filenames, +and source labels serve presentation and provenance. The exact input-error code +mapping is proposed under Open Decisions. + +The selected outcome retains the original returned cause, including ordinary +wrapping and joined errors, so supported `errors.Is` and `errors.As` queries +continue to work. Existing explanations, source context, diagnostic visibility, +and operation-specific reporting lifecycles apply to the newly classified +failures. Shared rendering and delivery failures follow the +[core output-failure rules](../cli.md#output-failures), including their effect on +usage outcomes. + +Apply the classification to configuration acquisition failures throughout the +standard suites, including failures returned by application callbacks when they +carry the supported ConfigKit evidence. CLI syntax, resource lookup, request +execution, and application-specific error semantics retain their owning +contracts. The work concerns interpreting returned failure evidence at the CLI +boundary. + +## Work needed + +### 1. Define the configuration-failure classification + +- Inventory ConfigKit v1.2.0's public diagnostic codes and their documented + meaning. Record the selected input-error cases explicitly using upstream + constants. Review all code families, including composition, imports, + inheritance, versions, overrides, secrets, and configured limits. +- Use error-severity diagnostics as failure evidence. Warning records can + accompany a failure or a successful result. +- Handle wrapped `LoadError` values and standard joined error branches. A failure + selects usage only when its failing evidence is classified as invalid input. + An operational or unclassified failing branch makes the combined result + operational. Preserve the original cause tree. +- Account for cancellation and deadlines before choosing usage. When an existing + checkpoint observes cancellation alongside an acquisition failure, preserve + both causes using the established error-joining approach. +- Keep classification within integration ownership, using the sharing approach + selected below. Existing core outcome constructors and reporting provide the + required mechanics. + +### 2. Apply it throughout ckconfig + +Wire the classification into acquisition failure handling for `config show`, +`config sources`, `config validate`, `config explain`, and `config diff`. + +Follow each operation's current preparation, aggregation, and reporting flow. +In particular, source and validation reports can accompany a failed load. +Detailed validation filters control displayed findings, while the complete +underlying failure supplies classification evidence. Comparison must classify a +failure on either loading side and retain the side-specific diagnostic context. + +Include native ConfigKit scopes, PromptKit configuration scopes, and custom +scope callbacks returning the supported evidence. Scope selection and input +preflight continue through their existing contracts. + +### 3. Apply it throughout ckprompt + +Cover engine-acquisition failures in catalogs, named prompt/profile/backend +inspection, requirements discovery, concrete request validation, prepared-request +preview, individual smoke operations, and aggregate smoke testing. + +Acquisition evidence may arrive through loaders or application request builders. +Use the same classification policy wherever it reaches the standard command's +failure boundary. For joined aggregate smoke errors, an operational or +unclassified failure takes precedence over input failures. Follow the existing +aggregate execution and report lifecycle when selecting the final outcome. + +### 4. Verify observable integration behavior + +Extend focused public-command tests where they provide regression protection. +Use upstream errors as fixtures to establish CommandKit's classification and +preservation behavior, following the +[integration testing policy](../policy/testing.md#upstream-integration-testing-ownership). + +Cover these distinct risks with a proportionate set of cases. + +- Malformed main and companion configuration produce status `2` with + `yaml.invalid` and the typed ConfigKit cause available through wrapping. +- Representative decoding and selected semantic-validation errors receive the + intended classification. Operational and unrecognized loading diagnostics + receive status `1`, including mixed failing evidence and warnings alongside + errors. +- The five configuration operations and affected PromptKit command families use + the policy, including comparison-side failures and aggregate smoke outcomes. +- Cancellation and deadlines receive status `1` and preserve their supported + identity. Already-canceled invocations retain the reviewed empty-output + behavior. A failed acquisition combined with cancellation preserves both causes. +- Source and validation reports retain their operation-specific failure evidence. + Detailed display filters do not determine failure classification. +- Output failures continue to follow the shared reporting contract when the + command has selected usage status `2`. + +Run the complete deterministic suites independently for each changed integration +and with the aligned local workspace as described in the +[developer guide](../development.md#documentation-validation). Exercise the +maintained examples where their existing assertions cover affected outcomes. +The Audita reproduction supplies supporting consumer evidence. CommandKit-owned +tests provide the maintained regression coverage. + +### 5. Document and prepare the integration releases + +Update the CLI status table and affected operation sections with the implemented +classification. Update any affected exported GoDoc and consumer guidance under +their canonical owners. Explain the classification boundary and fallback through +the chosen API's documentation. + +Prepare ckconfig v0.8.0 followed by ckprompt v0.8.0, with ckprompt selecting the +published ckconfig version. The status-contract change warrants minor releases +under the [compatibility policy](../policy/release.md#versioning-and-compatibility). +The release notes should explain that identifiable invalid configuration now +selects status `2`, alongside the operational classification and preserved error +evidence. Follow the [release procedure](../release.md) when publication is +authorized. Implementation completion and publication are tracked separately. + +## Completion evidence + +Implementation is complete when the accepted classification is exercised through +the public integration commands, the focused regression cases and selected +integration suites pass, and the canonical documentation describes the +implemented behavior. Review the final diff for the agreed scope and retire this +roadmap under the normal documentation lifecycle after recording any unfinished +release work. + +## Open Decisions + +### Exact diagnostic-code mapping + +**Recommendation.** Use an explicit mapping based on ConfigKit v1.2.0's public +code semantics. Include YAML-content and decoding errors, explicit +`version.error`, `resolve.error`, and `validate.error` findings, and other codes +whose documented meaning clearly identifies invalid configuration content. +Classify schema/options defects, acquisition failures, hook execution failures, +invalid callback paths or records, and validation mutation as operational. +Keep ambiguous limit, secret, and source-selection cases operational until their +individual meaning supports input classification. Review and record those cases +when preparing the implementation plan. + +This uses the evidence upstream already supplies and distinguishes reported +configuration rejection from callback machinery failing. Source evidence +alone may be insufficient to establish whether application-supplied data came +from an operator, so the rule follows the meaning recorded in upstream +diagnostics. + +**Alternative.** Add an application-supplied classification hook for cases that +depend on source ownership or application policy, alongside the built-in mapping. +Applications can then identify input failures using knowledge their loaders +possess. This supports genuinely contextual cases, but expands the public +integration contracts and creates additional configuration and precedence rules. +The current reproduction is addressable through existing typed diagnostics, so +the additional hook has limited demonstrated value in this scope. + +### Shared classification interface + +**Recommendation.** Place one narrowly documented classification function in +ckconfig and have ckprompt call it through its existing dependency. Keep command +explanation construction and reporting in their owning handlers. Select its +exact signature during implementation planning based on the needed cause and +cancellation inputs. A shared implementation reduces drift across the two +standard families and fits ckconfig's ownership of ConfigKit adaptation. The +public function adds an API maintenance obligation. + +**Alternative.** Maintain small private classifiers in both integration modules, +with matching behavioral cases. This keeps the helper private and supports +package-local evolution, but duplicates the diagnostic mapping and joined-error +rules. Future upstream code changes would require coordinated edits in both +modules. Prefer this approach if the shared signature would expose more policy +or machinery than the demonstrated integration need warrants.