eric/audita
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

Enable prompt preview validation and smoke commands

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-10 16:59:28 +00:00
parent 8cc298d092
commit 8e1e073731
4 changed files with 184 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
docs/internal/prompts.md
View File

@@ -1,37 +1,29 @@
# Audita Prompt Registry
`internal/prompts` embeds seven PromptKit YAML definitions under
`assets/prompts`. The IDs and selected version remain:
`internal/prompts` embeds seven PromptKit definitions in the `audita`
collection. The four proposal IDs are `audita/modules.glossary.proposal`,
`audita/modules.homophones.proposal`, `audita/modules.spoken_word.proposal`,
and `audita/modules.grammar.proposal`; the three validator IDs are
`audita/validators.spoken_form_plausibility`,
`audita/validators.meaning_reversal_review`, and
`audita/validators.editorial_review`.
- `modules.glossary.proposal` (`v1`)
- `modules.homophones.proposal` (`v1`)
- `modules.spoken_word.proposal` (`v1`)
- `modules.grammar.proposal` (`v1`)
- `validators.spoken_form_plausibility` (`v1`)
- `validators.meaning_reversal_review` (`v1`)
- `validators.editorial_review` (`v1`)
PromptKit composes embedded defaults with operator prompt sources. Shared
instructions come from the embedded fragments in `assets/shared`; the request
adapter fills absent `HardeningPolicy`, `ProposalContract`, and
`ValidationContract` variables without replacing values explicitly supplied by
the caller. Processing supplies Audita's response-schema contract separately.
The operator's prompt definition controls which variables its template uses.
For a standard request preview or validation, select a native PromptKit profile
and supply the template variables used by the selected prompt. The shipped
proposal definitions use `TranscriptDescriptionBlock`, `GlossaryJSON`, and
`SectionJSON`; the validator definitions use `TranscriptDescriptionBlock` and
`PayloadJSON`. The latter is supplied candidate JSON and does not require a
proposal inference call. An operator replacement may use different variables.
See the [upstream CLI reference](https://forge.maximumdirect.net/base-layer/commandkit/src/commit/f369d6a78253b6e6fcb1e023fb2e9c3af1d20ff0/docs/cli.md)
for the standard preview, validation, and smoke commands.
PromptKit reads the embedded definitions as shipped defaults and renders their
messages from named string variables. Operator documents in `profiles_dir` may
contribute native keyed prompt definitions and override an embedded
prompt/version, including its messages and definition-level output settings.
The effective operator text is captured when the runtime is constructed; the
embedded assets remain the source of Audita's shipped task instructions. The
role-bound adapter supplies
`HardeningPolicy` alongside `ProposalContract` and `ValidationContract` from
embedded fragments under `assets/shared`, after caller variables so callers
cannot replace those instructions. Common literal-span and identity guidance
lives in the proposal fragment; shared candidate-reference, context, and
confidence guidance lives in the validation fragment. Module-specific policies
and examples remain in YAML. Proposal prompts receive the section JSON, glossary
JSON where used, and transcript description block. Validator prompts receive
the correction payload and description block.
The embedded prompt assets retain the shipped task instructions and
output-schema association. An operator override controls which template
variables its messages reference: supplying a shared variable does not append
its text when the effective message omits that reference. The role-bound runtime
supplies the complete JSON Schema output contract and repair count; the
definition does not own an operator-facing repair default.
The correction-set schema requires a string replacement and accepts an empty
string. Proposal guidance uses it for eligible span deletion and permits literal
spacing edits; validator guidance evaluates the full before/after segment under
@@ -43,10 +35,9 @@ rejection, not a structural schema error.
filesystem to PromptKit and retains Audita's schema-key lookup. PromptKit's
prepared execution supplies definition and rendered hashes; Audita does not
recompute or promise equality with hashes from the removed template renderer.
The small Audita registry records only prompt ID, version, source, and embedded
path. Prepared runtime metadata belongs in diagnostics rather than the registry.
The small Audita registry records only prompt ID, source, and embedded path. Prepared runtime metadata belongs in diagnostics rather than the registry.
The structured-call contract carries prompt ID/version, string variables,
The structured-call contract carries prompt ID, string variables,
response-schema key, and logical-call attribution. Module and validator callers
pass these values through the role-bound adapter. See the
[LLM runtime](llm-runtime.md) for runtime assembly and

5
docs/roadmap/implementation.md
View File

@@ -275,6 +275,11 @@ for inspection or overriding native grammar or status behavior.
## Stage 6: Enable standard PromptKit request and smoke commands
**Completed.** Native preview, validation, and the complete smoke surface are
installed through the shared loader. Controlled transport checks cover operator
replacement, validator input, explicit smoke, and aggregate smoke. The processing
handler and maintained example projects are assigned below.
Own the remaining `internal/cli` adapters. Use
[the resolved command scope](commandkit.md#request-preview-validation-and-smoke-commands--resolved)
and `WithRequestPreparation`, `WithRequestValidation`, and `WithSmokeTesting`

40
internal/cli/app_wiring.go
View File

@@ -3,11 +3,13 @@ package cli
import (
"context"
"fmt"
"net/http"
"reflect"
"strings"
"gitea.maximumdirect.net/eric/audita/v3/internal/core/config"
"gitea.maximumdirect.net/eric/audita/v3/internal/framework/llm"
"gitea.maximumdirect.net/eric/audita/v3/internal/prompts"
"go.maximumdirect.net/ckconfig"
"go.maximumdirect.net/ckprompt"
commandkit "go.maximumdirect.net/commandkit"
@@ -16,7 +18,10 @@ import (
"go.maximumdirect.net/releasekit/buildinfo"
)
type invocationState struct{ engines []engineEntry }
type invocationState struct {
engines []engineEntry
client *http.Client
}
type engineEntry struct {
selection ckconfig.Selection
engine *pk.Engine
@@ -50,7 +55,11 @@ func loadPromptEngine(ctx context.Context, selection ckconfig.Selection, input s
if err != nil {
return nil, err
}
runtime, err := llm.NewRuntime(ctx, loaded, nil)
var client *http.Client
if state != nil {
client = state.client
}
runtime, err := llm.NewRuntime(ctx, loaded, client)
if err != nil {
return nil, err
}
@@ -61,14 +70,37 @@ func loadPromptEngine(ctx context.Context, selection ckconfig.Selection, input s
}
func buildApplication(extra ...commandkit.Declaration) (*commandkit.App, error) {
return buildApplicationWithClient(nil, extra...)
}
func buildApplicationWithClient(client *http.Client, extra ...commandkit.Declaration) (*commandkit.App, error) {
appScope := ckconfig.FromConfigKit("app", "Audita application settings.", loadApplication)
appScope.Features.Overrides = true
promptScope := ckprompt.ConfigScope(loadPromptEngine)
configCommand := ckconfig.Command(ckconfig.Suite[struct{}, *invocationState]{NewState: func() *invocationState { return &invocationState{} }, Scopes: []ckconfig.Scope[struct{}, *invocationState]{appScope, promptScope}})
configCommand := ckconfig.Command(ckconfig.Suite[struct{}, *invocationState]{NewState: func() *invocationState { return &invocationState{client: client} }, Scopes: []ckconfig.Scope[struct{}, *invocationState]{appScope, promptScope}})
definitions := ckprompt.WithNamespaces(ckprompt.FromEngine(loadPromptEngine), pk.Namespaces{})
definitions.NewState = func() *invocationState { return &invocationState{} }
definitions.NewState = func() *invocationState { return &invocationState{client: client} }
definitions = ckprompt.WithRequestValidation(definitions, loadPromptEngine, func(_ context.Context, seed ckprompt.ValidationRequest, _ struct{}, _ *invocationState) (pk.RunRequest, error) {
return applicationPromptRequest(seed.Request, seed.Namespaces), nil
})
definitions = ckprompt.WithRequestPreparation(definitions, loadPromptEngine, func(_ context.Context, seed ckprompt.PreparationRequest, _ struct{}, _ *invocationState) (pk.RunRequest, error) {
return applicationPromptRequest(seed.Request, seed.Namespaces), nil
})
definitions = ckprompt.WithSmokeTesting(definitions, loadPromptEngine, nil)
declarations := []commandkit.Declaration{configCommand, commandkit.Completion("audita"), processCommand()}
declarations = append(declarations, ckprompt.Declarations(definitions)...)
declarations = append(declarations, extra...)
return commandkit.New(commandkit.Identity{Name: "audita", Version: buildinfo.Version()}, declarations...)
}
// applicationPromptRequest supplies instructions only for Audita's maintained prompts.
func applicationPromptRequest(request pk.RunRequest, namespaces *pk.Namespaces) pk.RunRequest {
id := request.PromptID
if !strings.Contains(id, "/") && namespaces != nil && namespaces.Prompt != "" {
id = namespaces.Prompt + "/" + id
}
if _, ok := prompts.LookupMetadata(id); ok {
request.Vars = prompts.WithSharedVariables(request.Vars)
}
return request
}

119
internal/cli/app_wiring_test.go
View File

@@ -3,12 +3,16 @@ package cli
import (
"bytes"
"context"
"encoding/json"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
commandkit "go.maximumdirect.net/commandkit"
pk "go.maximumdirect.net/promptkit"
)
func TestNativeConfigurationAndResourceCommands(t *testing.T) {
@@ -75,3 +79,118 @@ func TestExplicitEmptyConfigurationPathFails(t *testing.T) {
t.Fatal("explicit empty path accepted")
}
}
func TestRequestInspectionAndExplicitSmokeUseSameSources(t *testing.T) {
dir := t.TempDir()
file := filepath.Join(dir, "config.yml")
if err := os.WriteFile(file, []byte("version: 3\n"), 0644); err != nil {
t.Fatal(err)
}
resources := `backends:
- id: local
endpoint: https://example.invalid/v1
smoke_tests:
default_profile: fixture
authentication:
kind: none
profiles:
- id: fixture
backend: local
model: sample
collections:
audita:
prompts:
- id: modules.grammar.proposal
messages:
- role: user
content: 'Replacement {{.HardeningPolicy}} {{.SectionJSON}}'
output:
format: text
validation_mode: none
`
if err := os.WriteFile(filepath.Join(dir, "prompts.yml"), []byte(resources), 0644); err != nil {
t.Fatal(err)
}
calls := 0
client := &http.Client{Transport: roundTripFunc(func(req *http.Request) (*http.Response, error) {
calls++
body := `{}`
if req.Method == http.MethodPost {
requestBody, err := io.ReadAll(req.Body)
if err != nil {
t.Fatal(err)
}
content := "PROMPTKIT_OK"
if strings.Contains(string(requestBody), "response_format") {
content = `{"status":"ok"}`
}
encoded, err := json.Marshal(content)
if err != nil {
t.Fatal(err)
}
body = `{"model":"sample","choices":[{"message":{"content":` + string(encoded) + `}}]}`
}
return &http.Response{StatusCode: 200, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(body)), Request: req}, nil
})}
app, err := buildApplicationWithClient(client)
if err != nil {
t.Fatal(err)
}
execute := func(args ...string) (commandkit.Outcome, string, string) {
t.Helper()
var out, stderr bytes.Buffer
result := app.Execute(context.Background(), args, commandkit.Streams{Out: &out, Err: &stderr})
return result, out.String(), stderr.String()
}
common := []string{"--config", file, "--llm-profile", "fixture"}
args := append([]string{"promptkit", "show", "prepared", "audita/modules.grammar.proposal"}, common...)
args = append(args, "--llm-var", "SectionJSON=[]", "--llm-var", "HardeningPolicy=custom-instruction")
outcome, out, stderr := execute(args...)
if outcome.Status != 0 || !strings.Contains(out, "Replacement custom-instruction") {
t.Fatalf("preview status=%d out=%q err=%q", outcome.Status, out, stderr)
}
args = append([]string{"promptkit", "requirements", "validate", "audita/validators.editorial_review"}, common...)
args = append(args, "--llm-var", `PayloadJSON=[{"correction_index":1,"original_segment_text":"teh cat","corrected_segment_text":"the cat"}]`, "--llm-var", "TranscriptDescriptionBlock=")
outcome, out, stderr = execute(args...)
if outcome.Status != 0 {
t.Fatalf("validator status=%d out=%q err=%q", outcome.Status, out, stderr)
}
validatorPreview := append([]string{"promptkit", "show", "prepared", "audita/validators.editorial_review"}, common...)
validatorPreview = append(validatorPreview, "--llm-var", `PayloadJSON=[{"correction_index":1,"original_segment_text":"teh cat","corrected_segment_text":"the cat"}]`, "--llm-var", "TranscriptDescriptionBlock=")
outcome, out, stderr = execute(validatorPreview...)
if outcome.Status != 0 || !strings.Contains(out, "teh cat") {
t.Fatalf("validator preview status=%d out=%q err=%q", outcome.Status, out, stderr)
}
if calls != 0 {
t.Fatalf("inspection made %d provider calls", calls)
}
outcome, out, stderr = execute("promptkit", "test", "reachability", "--config", file, "--llm-backend", "local")
if outcome.Status != 0 || calls == 0 {
t.Fatalf("smoke status=%d calls=%d out=%q err=%q", outcome.Status, calls, out, stderr)
}
previous := calls
outcome, out, stderr = execute("promptkit", "test", "all", "--config", file, "--llm-backend", "local", "--llm-profile", "fixture", "--format=json")
if outcome.Status != 0 || calls <= previous || !strings.Contains(out, "structured_output") || !strings.Contains(out, "text_inference") {
t.Fatalf("aggregate status=%d calls=%d out=%q err=%q", outcome.Status, calls, out, stderr)
}
}
type roundTripFunc func(*http.Request) (*http.Response, error)
func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }
func TestRequestDefaultsRespectNamespaceAndCallerVariables(t *testing.T) {
request := pk.RunRequest{PromptID: "modules.grammar.proposal", Vars: map[string]string{"HardeningPolicy": "caller"}}
result := applicationPromptRequest(request, &pk.Namespaces{Prompt: "audita"})
if result.Vars["HardeningPolicy"] != "caller" || result.Vars["ProposalContract"] == "" {
t.Fatalf("variables=%v", result.Vars)
}
if _, changed := request.Vars["ProposalContract"]; changed {
t.Fatal("input variables mutated")
}
generic := pk.RunRequest{PromptID: "generic", Vars: map[string]string{"x": "y"}}
result = applicationPromptRequest(generic, nil)
if len(result.Vars) != 1 || result.Vars["x"] != "y" {
t.Fatalf("generic request changed: %v", result.Vars)
}
}