Plan invalid configuration outcome handling
This commit is contained in:
237
docs/roadmap/invalid-configuration-outcomes.md
Normal file
237
docs/roadmap/invalid-configuration-outcomes.md
Normal file
@@ -0,0 +1,237 @@
|
||||
# Invalid configuration outcomes
|
||||
|
||||
Status — proposed implementation roadmap. The user has accepted the targeted
|
||||
integration behavior. The decisions at the end identify remaining policy and
|
||||
API choices for the implementation plan.
|
||||
|
||||
## Purpose
|
||||
|
||||
Make standard ckconfig and ckprompt commands return usage status `2` for
|
||||
identifiable invalid configuration input, while returning operational status `1`
|
||||
for cancellation and other operational failures. Preserve actionable upstream
|
||||
diagnostics and supported typed error identity through the integration boundary.
|
||||
|
||||
CommandKit owns this CLI classification. ConfigKit and PromptKit own loading,
|
||||
validation, diagnostic meaning, and their public error contracts. Application
|
||||
commands own their domain outcomes. These responsibilities follow the
|
||||
[architecture policy](../policy/architecture.md#ownership-boundaries).
|
||||
|
||||
## Current baseline
|
||||
|
||||
The reviewed baseline is CommandKit commit
|
||||
`1e92a87706e44684ee34395245e5b633474677e0`, following the publication of
|
||||
ckconfig v0.7.0 and ckprompt v0.7.0. Both integration modules select core
|
||||
v0.6.0 and ConfigKit v1.2.0. ckprompt selects ckconfig v0.7.0 and PromptKit
|
||||
v0.21.0. Their manifests own the dependency selections.
|
||||
|
||||
The consumer report reproduced malformed operator YAML in Audita, including
|
||||
its automatic `prompts.yml` companion and an explicitly selected application
|
||||
configuration. Delegated inspection returned `yaml.invalid` with status `1`,
|
||||
while Audita's processing command returned the same diagnostic with status `2`.
|
||||
The report's reproduction is retained in Audita's
|
||||
[upstream handoff](https://forge.maximumdirect.net/eric/audita/src/commit/a355f05a273d6ac9d577eab8d30caa60cb74eee8/docs/roadmap/implementation.md).
|
||||
|
||||
The review confirmed the behavior through Audita's existing regression tests and
|
||||
a temporary public-API reproduction against the aligned CommandKit workspace.
|
||||
The latter covered catalog listing and all five configuration operations in both
|
||||
the application and PromptKit scopes, including wrapped ConfigKit loading errors
|
||||
and already-canceled contexts.
|
||||
|
||||
- [FromConfigKit](../../integrations/ckconfig/configkit_adapter.go) and
|
||||
[the PromptKit adapters](../../integrations/ckprompt/adapters.go) preserve the
|
||||
ConfigKit loading error and adapt its diagnostic evidence.
|
||||
- [configFailure](../../integrations/ckconfig/operations.go) and
|
||||
[definitionFailure](../../integrations/ckprompt/command.go) construct operational
|
||||
outcomes for the affected failures. Configuration comparison and validation
|
||||
also reach the former helper.
|
||||
- ConfigKit's public `LoadError` represents several kinds of loading failure,
|
||||
including malformed YAML, acquisition errors, and schema or callback problems.
|
||||
Its public diagnostic codes provide the classification evidence. Its error
|
||||
contract intentionally exposes safe diagnostics and preserves its own typed
|
||||
identity across CommandKit adaptation.
|
||||
- `config sources` can report discovered source evidence after failed loading,
|
||||
and `config validate` can report failed validation. Cancellation observed
|
||||
before acquisition leaves stdout empty in the reviewed command paths.
|
||||
|
||||
The current [CLI contract](../cli.md#outcome-status) assigns loading failures
|
||||
status `1`, with more specific statements in the configuration comparison,
|
||||
requirements, validation, and smoke sections. This roadmap proposes a deliberate
|
||||
change to that published status behavior.
|
||||
|
||||
## Targeted end state
|
||||
|
||||
| Evidence available to the integration | Selected outcome |
|
||||
| --- | --- |
|
||||
| ConfigKit error diagnostics that identify invalid configuration input under the selected mapping | Usage, status `2` |
|
||||
| Caller cancellation or deadline observed at an existing operation checkpoint, or present in the returned error chain | Operational, status `1` |
|
||||
| I/O, schema or wiring defects, callback execution failures, and other recognized operational failures | Operational, status `1` |
|
||||
| An unrecognized failure, including a loading error with insufficient classification evidence | Operational, status `1` |
|
||||
| Successful acquisition with warnings | Follow the command's successful-result behavior |
|
||||
|
||||
Classification uses upstream structured evidence. Diagnostic text, filenames,
|
||||
and source labels serve presentation and provenance. The exact input-error code
|
||||
mapping is proposed under Open Decisions.
|
||||
|
||||
The selected outcome retains the original returned cause, including ordinary
|
||||
wrapping and joined errors, so supported `errors.Is` and `errors.As` queries
|
||||
continue to work. Existing explanations, source context, diagnostic visibility,
|
||||
and operation-specific reporting lifecycles apply to the newly classified
|
||||
failures. Shared rendering and delivery failures follow the
|
||||
[core output-failure rules](../cli.md#output-failures), including their effect on
|
||||
usage outcomes.
|
||||
|
||||
Apply the classification to configuration acquisition failures throughout the
|
||||
standard suites, including failures returned by application callbacks when they
|
||||
carry the supported ConfigKit evidence. CLI syntax, resource lookup, request
|
||||
execution, and application-specific error semantics retain their owning
|
||||
contracts. The work concerns interpreting returned failure evidence at the CLI
|
||||
boundary.
|
||||
|
||||
## Work needed
|
||||
|
||||
### 1. Define the configuration-failure classification
|
||||
|
||||
- Inventory ConfigKit v1.2.0's public diagnostic codes and their documented
|
||||
meaning. Record the selected input-error cases explicitly using upstream
|
||||
constants. Review all code families, including composition, imports,
|
||||
inheritance, versions, overrides, secrets, and configured limits.
|
||||
- Use error-severity diagnostics as failure evidence. Warning records can
|
||||
accompany a failure or a successful result.
|
||||
- Handle wrapped `LoadError` values and standard joined error branches. A failure
|
||||
selects usage only when its failing evidence is classified as invalid input.
|
||||
An operational or unclassified failing branch makes the combined result
|
||||
operational. Preserve the original cause tree.
|
||||
- Account for cancellation and deadlines before choosing usage. When an existing
|
||||
checkpoint observes cancellation alongside an acquisition failure, preserve
|
||||
both causes using the established error-joining approach.
|
||||
- Keep classification within integration ownership, using the sharing approach
|
||||
selected below. Existing core outcome constructors and reporting provide the
|
||||
required mechanics.
|
||||
|
||||
### 2. Apply it throughout ckconfig
|
||||
|
||||
Wire the classification into acquisition failure handling for `config show`,
|
||||
`config sources`, `config validate`, `config explain`, and `config diff`.
|
||||
|
||||
Follow each operation's current preparation, aggregation, and reporting flow.
|
||||
In particular, source and validation reports can accompany a failed load.
|
||||
Detailed validation filters control displayed findings, while the complete
|
||||
underlying failure supplies classification evidence. Comparison must classify a
|
||||
failure on either loading side and retain the side-specific diagnostic context.
|
||||
|
||||
Include native ConfigKit scopes, PromptKit configuration scopes, and custom
|
||||
scope callbacks returning the supported evidence. Scope selection and input
|
||||
preflight continue through their existing contracts.
|
||||
|
||||
### 3. Apply it throughout ckprompt
|
||||
|
||||
Cover engine-acquisition failures in catalogs, named prompt/profile/backend
|
||||
inspection, requirements discovery, concrete request validation, prepared-request
|
||||
preview, individual smoke operations, and aggregate smoke testing.
|
||||
|
||||
Acquisition evidence may arrive through loaders or application request builders.
|
||||
Use the same classification policy wherever it reaches the standard command's
|
||||
failure boundary. For joined aggregate smoke errors, an operational or
|
||||
unclassified failure takes precedence over input failures. Follow the existing
|
||||
aggregate execution and report lifecycle when selecting the final outcome.
|
||||
|
||||
### 4. Verify observable integration behavior
|
||||
|
||||
Extend focused public-command tests where they provide regression protection.
|
||||
Use upstream errors as fixtures to establish CommandKit's classification and
|
||||
preservation behavior, following the
|
||||
[integration testing policy](../policy/testing.md#upstream-integration-testing-ownership).
|
||||
|
||||
Cover these distinct risks with a proportionate set of cases.
|
||||
|
||||
- Malformed main and companion configuration produce status `2` with
|
||||
`yaml.invalid` and the typed ConfigKit cause available through wrapping.
|
||||
- Representative decoding and selected semantic-validation errors receive the
|
||||
intended classification. Operational and unrecognized loading diagnostics
|
||||
receive status `1`, including mixed failing evidence and warnings alongside
|
||||
errors.
|
||||
- The five configuration operations and affected PromptKit command families use
|
||||
the policy, including comparison-side failures and aggregate smoke outcomes.
|
||||
- Cancellation and deadlines receive status `1` and preserve their supported
|
||||
identity. Already-canceled invocations retain the reviewed empty-output
|
||||
behavior. A failed acquisition combined with cancellation preserves both causes.
|
||||
- Source and validation reports retain their operation-specific failure evidence.
|
||||
Detailed display filters do not determine failure classification.
|
||||
- Output failures continue to follow the shared reporting contract when the
|
||||
command has selected usage status `2`.
|
||||
|
||||
Run the complete deterministic suites independently for each changed integration
|
||||
and with the aligned local workspace as described in the
|
||||
[developer guide](../development.md#documentation-validation). Exercise the
|
||||
maintained examples where their existing assertions cover affected outcomes.
|
||||
The Audita reproduction supplies supporting consumer evidence. CommandKit-owned
|
||||
tests provide the maintained regression coverage.
|
||||
|
||||
### 5. Document and prepare the integration releases
|
||||
|
||||
Update the CLI status table and affected operation sections with the implemented
|
||||
classification. Update any affected exported GoDoc and consumer guidance under
|
||||
their canonical owners. Explain the classification boundary and fallback through
|
||||
the chosen API's documentation.
|
||||
|
||||
Prepare ckconfig v0.8.0 followed by ckprompt v0.8.0, with ckprompt selecting the
|
||||
published ckconfig version. The status-contract change warrants minor releases
|
||||
under the [compatibility policy](../policy/release.md#versioning-and-compatibility).
|
||||
The release notes should explain that identifiable invalid configuration now
|
||||
selects status `2`, alongside the operational classification and preserved error
|
||||
evidence. Follow the [release procedure](../release.md) when publication is
|
||||
authorized. Implementation completion and publication are tracked separately.
|
||||
|
||||
## Completion evidence
|
||||
|
||||
Implementation is complete when the accepted classification is exercised through
|
||||
the public integration commands, the focused regression cases and selected
|
||||
integration suites pass, and the canonical documentation describes the
|
||||
implemented behavior. Review the final diff for the agreed scope and retire this
|
||||
roadmap under the normal documentation lifecycle after recording any unfinished
|
||||
release work.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
### Exact diagnostic-code mapping
|
||||
|
||||
**Recommendation.** Use an explicit mapping based on ConfigKit v1.2.0's public
|
||||
code semantics. Include YAML-content and decoding errors, explicit
|
||||
`version.error`, `resolve.error`, and `validate.error` findings, and other codes
|
||||
whose documented meaning clearly identifies invalid configuration content.
|
||||
Classify schema/options defects, acquisition failures, hook execution failures,
|
||||
invalid callback paths or records, and validation mutation as operational.
|
||||
Keep ambiguous limit, secret, and source-selection cases operational until their
|
||||
individual meaning supports input classification. Review and record those cases
|
||||
when preparing the implementation plan.
|
||||
|
||||
This uses the evidence upstream already supplies and distinguishes reported
|
||||
configuration rejection from callback machinery failing. Source evidence
|
||||
alone may be insufficient to establish whether application-supplied data came
|
||||
from an operator, so the rule follows the meaning recorded in upstream
|
||||
diagnostics.
|
||||
|
||||
**Alternative.** Add an application-supplied classification hook for cases that
|
||||
depend on source ownership or application policy, alongside the built-in mapping.
|
||||
Applications can then identify input failures using knowledge their loaders
|
||||
possess. This supports genuinely contextual cases, but expands the public
|
||||
integration contracts and creates additional configuration and precedence rules.
|
||||
The current reproduction is addressable through existing typed diagnostics, so
|
||||
the additional hook has limited demonstrated value in this scope.
|
||||
|
||||
### Shared classification interface
|
||||
|
||||
**Recommendation.** Place one narrowly documented classification function in
|
||||
ckconfig and have ckprompt call it through its existing dependency. Keep command
|
||||
explanation construction and reporting in their owning handlers. Select its
|
||||
exact signature during implementation planning based on the needed cause and
|
||||
cancellation inputs. A shared implementation reduces drift across the two
|
||||
standard families and fits ckconfig's ownership of ConfigKit adaptation. The
|
||||
public function adds an API maintenance obligation.
|
||||
|
||||
**Alternative.** Maintain small private classifiers in both integration modules,
|
||||
with matching behavioral cases. This keeps the helper private and supports
|
||||
package-local evolution, but duplicates the diagnostic mapping and joined-error
|
||||
rules. Future upstream code changes would require coordinated edits in both
|
||||
modules. Prefer this approach if the shared signature would expose more policy
|
||||
or machinery than the demonstrated integration need warrants.
|
||||
Reference in New Issue
Block a user