Complete resource inspection validation and coverage

This commit is contained in:
2026-11-06 15:34:21 +00:00
parent 19a0604874
commit f083ad5309
6 changed files with 61 additions and 19 deletions

View File

@@ -6,11 +6,11 @@ Implement the accepted [resource inspection roadmap](inspect-backend.md). That
document owns intent, scope, and product decisions; this plan owns the concrete document owns intent, scope, and product decisions; this plan owns the concrete
steps and representation choices needed to deliver them. steps and representation choices needed to deliver them.
Baseline: `951739e`. ConfigKit v1.1.0 composes and resolves definitions once. At baseline `951739e`, ConfigKit v1.1.0 composed and resolved definitions once.
Promptkit captures executable prompts and backend-completed profile targets. Promptkit captured executable prompts and backend-completed profile targets.
`InspectPrompt` currently exposes limited metadata; `InspectProfile` exposes an `InspectPrompt` exposed limited metadata; `InspectProfile` exposed an
execution target. There is no `InspectBackend` or typed `PromptSource`, and execution target. There was no `InspectBackend` or typed `PromptSource`, and
`BackendConfig` lacks `Extends`. Existing inspection values do not have the `BackendConfig` lacked `Extends`. Existing inspection values did not have the
selected generic JSON/YAML contract. selected generic JSON/YAML contract.
Each numbered stage is intended for one Luna implementation prompt, in order. Each numbered stage is intended for one Luna implementation prompt, in order.
@@ -571,6 +571,13 @@ ConfigKit composition contracts intact.
## Stage 12: Validate And Close The Sprint ## Stage 12: Validate And Close The Sprint
**Completed:** Audited the selected resource, encoding, source, ownership, and
null-migration boundaries; added coverage for inherited diagnostics and explicit
serialization values. With Go 1.27.3, `GOTOOLCHAIN=local`, and `GOWORK=off`,
`go test -race ./...`, `go vet ./...`, and module tidy verification pass. Changed
Go files are formatted, local documentation targets and whitespace are checked,
and the feature roadmap and plan are retained for review.
**Scope:** Combined verification and final cleanup of this feature only. **Scope:** Combined verification and final cleanup of this feature only.
1. Confirm that all three inspection operations, every selected result field, 1. Confirm that all three inspection operations, every selected result field,

View File

@@ -6,8 +6,8 @@ Help operators using CommandKit-based applications inspect the configured
prompt, profile, or backend selected by ID. CommandKit owns commands and report prompt, profile, or backend selected by ID. CommandKit owns commands and report
presentation; Promptkit exposes the selected resource's resolved configuration. presentation; Promptkit exposes the selected resource's resolved configuration.
The direction is accepted for planning; implementation has not begun. Selected The selected work has been implemented; this roadmap is retained for review.
work includes fuller prompt inspection, profile inspection of the resolved It includes fuller prompt inspection, profile inspection of the resolved
profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt
construction through `PromptConfig`/`PromptSource`, generically renderable construction through `PromptConfig`/`PromptSource`, generically renderable
inspection results, and simplification of Promptkit's post-handoff null handling. inspection results, and simplification of Promptkit's post-handoff null handling.
@@ -26,7 +26,7 @@ inspection assembles everything affecting execution.
| --- | --- | | --- | --- |
| `InspectPrompt` | Resolved prompt definition, including captured imported message content. | | `InspectPrompt` | Resolved prompt definition, including captured imported message content. |
| `InspectProfile` | Resolved profile definition, before backend-to-profile execution completion. | | `InspectProfile` | Resolved profile definition, before backend-to-profile execution completion. |
| Proposed `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. | | `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. |
For example, given: For example, given:
@@ -153,8 +153,8 @@ lookup. A captured prompt remains inspectable without a runnable default profile
## Resolved Profile Inspection ## Resolved Profile Inspection
This is a deliberate change to existing `InspectProfile` behavior. It currently This is a deliberate change to the former `InspectProfile` behavior. It previously
returns backend-completed `EffectiveModelParams`; the new resource boundary returned backend-completed `EffectiveModelParams`; the resource boundary
requires the resolved profile's own fields instead. requires the resolved profile's own fields instead.
Report fields supplied by the profile or inherited through ConfigKit, including Report fields supplied by the profile or inherited through ConfigKit, including
@@ -346,11 +346,11 @@ source round-tripping are not selected deliverables.
All five decisions below are resolved, including generic rendering and the All five decisions below are resolved, including generic rendering and the
post-handoff null simplification in decision 4. Numbering is retained for post-handoff null simplification in decision 4. Numbering is retained for
continuity with the discussion; implementation is pending. continuity with the discussion; the selected implementation is complete.
### 1. How Should An Unknown Backend Be Classified? ### 1. How Should An Unknown Backend Be Classified?
**Status:** Accepted; implementation pending. **Status:** Accepted and implemented.
**Decision:** Add public `ErrBackendNotFound` for an absent backend selected by **Decision:** Add public `ErrBackendNotFound` for an absent backend selected by
`InspectBackend`, following the existing prompt/profile sentinel pattern. `InspectBackend`, following the existing prompt/profile sentinel pattern.
@@ -391,7 +391,7 @@ where to place those additions remains necessary for resource inspection.
### 3. Should Typed Prompt Construction Be Added In This Round? ### 3. Should Typed Prompt Construction Be Added In This Round?
**Status:** Accepted; implementation pending. **Status:** Accepted and implemented.
**Decision:** Add `PromptConfig` and `PromptSource`, including inheritance, **Decision:** Add `PromptConfig` and `PromptSource`, including inheritance,
following the thin mapping-constructor pattern described above. Retain the following the thin mapping-constructor pattern described above. Retain the
@@ -411,7 +411,7 @@ is unnecessary; raw mappings continue to serve those cases.
### 4. How Should Resolved Resource Results And Presence Be Represented? ### 4. How Should Resolved Resource Results And Presence Be Represented?
**Status:** Accepted; implementation pending. **Status:** Accepted and implemented.
**Decision (Option A):** Use resource-oriented typed inspection results, reusing **Decision (Option A):** Use resource-oriented typed inspection results, reusing
suitable public nested values. Change `ProfileInspection` deliberately to report suitable public nested values. Change `ProfileInspection` deliberately to report

View File

@@ -51,3 +51,17 @@ func TestResolvedBackendAndProfileResources(t *testing.T) {
t.Fatalf("cancellation = %v", err) t.Fatalf("cancellation = %v", err)
} }
} }
func TestBackendResourcePreservesInheritedDiagnostics(t *testing.T) {
state, err := load(mapping("backend diagnostics", map[string]any{"backends": []any{
map[string]any{"id": "parent", "endpoint": "https://provider.test/v1", "smoke_tests": map[string]any{"reachability": map[string]any{"endpoint": "https://probe.test/ready"}, "authentication": map[string]any{"kind": "none"}}},
map[string]any{"id": "child", "extends": map[string]any{"id": "parent"}},
}}), nil)
if err != nil {
t.Fatal(err)
}
got, err := state.InspectBackend(context.Background(), "child")
if err != nil || got.ID != "child" || got.ConcurrencyLimit != nil || got.QueueCapacity != nil || got.SmokeTests == nil || got.SmokeTests.Reachability == nil || got.SmokeTests.Reachability.Endpoint == nil || *got.SmokeTests.Reachability.Endpoint != "https://probe.test/ready" || got.SmokeTests.Authentication == nil || got.SmokeTests.Authentication.Kind == nil || *got.SmokeTests.Authentication.Kind != "none" || got.SmokeTests.Authentication.Endpoint != nil {
t.Fatalf("inherited diagnostics = %+v, %v", got, err)
}
}

View File

@@ -92,3 +92,15 @@ func TestTypedPromptSourceDefersValidation(t *testing.T) {
t.Fatalf("invalid final source: %v", err) t.Fatalf("invalid final source: %v", err)
} }
} }
func TestNilTypedPromptSourceSuppliesEmptySequence(t *testing.T) {
source := promptkit.PromptSource("empty", nil)
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{Main: &configkit.Input{Source: source}}}, promptkit.WithBundledOpenRouter(false))
if err != nil {
t.Fatal(err)
}
prompts, err := engine.ListPrompts(context.Background())
if err != nil || len(prompts) != 0 {
t.Fatalf("empty source = %v, %v", prompts, err)
}
}

View File

@@ -42,4 +42,10 @@ func TestResourceInspectionSelections(t *testing.T) {
if _, err := engine.InspectBackend(canceled, "local"); !errors.Is(err, context.Canceled) { if _, err := engine.InspectBackend(canceled, "local"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled = %v", err) t.Fatalf("canceled = %v", err)
} }
if _, err := engine.InspectPrompt(canceled, "prompt"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled prompt = %v", err)
}
if _, err := engine.InspectProfile(canceled, "partial"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled profile = %v", err)
}
} }

View File

@@ -13,8 +13,8 @@ import (
func TestResourceInspectionGenericEncoding(t *testing.T) { func TestResourceInspectionGenericEncoding(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{Main: &configkit.Input{Source: configkit.Mapping("resources", map[string]any{ engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{Main: &configkit.Input{Source: configkit.Mapping("resources", map[string]any{
"backends": []any{map[string]any{"id": "local", "endpoint": "https://example.test/v1", "concurrency_limit": 2, "extra_params": map[string]any{"null": nil, "array": []any{nil}}, "smoke_tests": map[string]any{"reachability": map[string]any{}}}, map[string]any{"id": "plain", "endpoint": "https://plain.test/v1"}}, "backends": []any{map[string]any{"id": "local", "endpoint": "https://example.test/v1", "api_key_env": "BACKEND_KEY", "concurrency_limit": 2, "extra_params": map[string]any{"null": nil, "array": []any{nil}}, "smoke_tests": map[string]any{"reachability": map[string]any{}, "authentication": map[string]any{"kind": "none"}}}, map[string]any{"id": "plain", "endpoint": "https://plain.test/v1"}},
"profiles": []any{map[string]any{"id": "parent", "backend": "local", "model": "model"}, map[string]any{"id": "profile", "extends": map[string]any{"id": "parent"}, "temperature": 0, "api_key_required": false, "endpoint": "", "extra_params": map[string]any{}}, map[string]any{"id": "unset", "extra_params": nil}}, "profiles": []any{map[string]any{"id": "parent", "backend": "local", "model": "model"}, map[string]any{"id": "profile", "extends": map[string]any{"id": "parent"}, "temperature": 0, "top_p": 0, "max_tokens": 0, "timeout_seconds": 0, "service_tier": "", "reasoning_effort": "", "api_key_env": "", "api_key_required": false, "endpoint": "", "extra_params": map[string]any{}}, map[string]any{"id": "unset", "extra_params": nil}},
"prompts": []any{map[string]any{"id": "prompt", "inputs": []any{map[string]any{"name": "a", "required": false, "description": ""}, map[string]any{"name": "b"}}, "messages": []any{map[string]any{"role": "user", "content": "", "cache_control": map[string]any{"type": "ephemeral", "ttl": ""}}}, "output": map[string]any{"format": "text", "validation_mode": "basic", "repair_attempts": 0}}, map[string]any{"id": "empty-inputs", "inputs": []any{}, "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}, map[string]any{"id": "omitted-inputs", "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}}, "prompts": []any{map[string]any{"id": "prompt", "inputs": []any{map[string]any{"name": "a", "required": false, "description": ""}, map[string]any{"name": "b"}}, "messages": []any{map[string]any{"role": "user", "content": "", "cache_control": map[string]any{"type": "ephemeral", "ttl": ""}}}, "output": map[string]any{"format": "text", "validation_mode": "basic", "repair_attempts": 0}}, map[string]any{"id": "empty-inputs", "inputs": []any{}, "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}, map[string]any{"id": "omitted-inputs", "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}},
})}}}) })}}})
if err != nil { if err != nil {
@@ -77,7 +77,7 @@ func TestResourceInspectionGenericEncoding(t *testing.T) {
} }
} }
b := decode(backend) b := decode(backend)
if b["id"] != "local" || fmt.Sprint(b["concurrency_limit"]) != "2" || fmt.Sprint(b["queue_capacity"]) != "1024" { if b["id"] != "local" || b["api_key_env"] != "BACKEND_KEY" || fmt.Sprint(b["concurrency_limit"]) != "2" || fmt.Sprint(b["queue_capacity"]) != "1024" {
t.Errorf("backend = %#v", b) t.Errorf("backend = %#v", b)
} }
params := b["extra_params"].(map[string]any) params := b["extra_params"].(map[string]any)
@@ -85,16 +85,19 @@ func TestResourceInspectionGenericEncoding(t *testing.T) {
t.Errorf("provider nulls = %#v", params) t.Errorf("provider nulls = %#v", params)
} }
smoke := b["smoke_tests"].(map[string]any) smoke := b["smoke_tests"].(map[string]any)
if smoke["authentication"].(map[string]any)["kind"] != "none" {
t.Errorf("authentication = %#v", smoke)
}
if reach := smoke["reachability"].(map[string]any); len(reach) != 0 { if reach := smoke["reachability"].(map[string]any); len(reach) != 0 {
t.Errorf("reachability = %#v", reach) t.Errorf("reachability = %#v", reach)
} }
absent(b, "extends") absent(b, "extends")
absent(decode(plain), "smoke_tests", "queue_capacity", "concurrency_limit") absent(decode(plain), "smoke_tests", "queue_capacity", "concurrency_limit")
p := decode(profile) p := decode(profile)
if p["id"] != "profile" || p["backend"] != "local" || p["endpoint"] != "" || p["api_key_required"] != false || fmt.Sprint(p["temperature"]) != "0" || len(p["extra_params"].(map[string]any)) != 0 { if p["id"] != "profile" || p["backend"] != "local" || p["model"] != "model" || p["endpoint"] != "" || p["api_key_env"] != "" || p["service_tier"] != "" || p["reasoning_effort"] != "" || p["api_key_required"] != false || fmt.Sprint(p["temperature"]) != "0" || fmt.Sprint(p["top_p"]) != "0" || fmt.Sprint(p["max_tokens"]) != "0" || fmt.Sprint(p["timeout_seconds"]) != "0" || len(p["extra_params"].(map[string]any)) != 0 {
t.Errorf("profile = %#v", p) t.Errorf("profile = %#v", p)
} }
absent(p, "extends", "effective_model_params", "timeout_seconds", "api_key_env") absent(p, "extends", "effective_model_params")
absent(decode(unset), "extra_params", "endpoint") absent(decode(unset), "extra_params", "endpoint")
q := decode(prompt) q := decode(prompt)
inputs := q["inputs"].([]any) inputs := q["inputs"].([]any)