Resolve native prompt asset and schema design
This commit is contained in:
@@ -21,10 +21,11 @@ planning work. Implementation remains pending.
|
||||
|
||||
The roadmap resolves U10's CommandKit scope, tracks the pending upstream
|
||||
message-content hash in U1, replaces U8's checkpoint design, and narrows U9 to
|
||||
implementation details within the agreed output/debug contracts. Schema handling,
|
||||
role-specific correction/fallback policy, worker defaults, and concrete cache
|
||||
controls/transition remain visible there. Item-level directions below record the
|
||||
accepted design; the roadmap is the authority for sequencing and current status.
|
||||
implementation details within the agreed output/debug contracts. U2 now resolves
|
||||
asset and schema assembly. Role-specific correction/fallback policy, worker
|
||||
defaults, and concrete cache controls/transition remain visible there. Item-level
|
||||
directions below record the accepted design; the roadmap is the authority for
|
||||
sequencing and current status.
|
||||
|
||||
## Agreed Proposal
|
||||
|
||||
@@ -1213,8 +1214,9 @@ separate Phase 7 is retired.
|
||||
Its dependency baseline includes the selected versions and pending upstream hash
|
||||
request. U10 is resolved; U8 now concerns cache operation/transition, and U9
|
||||
concerns implementation details within the selected output/debug contracts.
|
||||
Earlier receipt/manifest inventories are retired. Remaining schema, correction,
|
||||
fallback, worker, and cache-transition choices are explicit in its
|
||||
Earlier receipt/manifest inventories are retired, and U2 resolves asset/schema
|
||||
assembly. Remaining correction, fallback, worker, and cache-transition choices
|
||||
are explicit in its
|
||||
[unresolved-choice table](promptkit.md#unresolved-choices).
|
||||
|
||||
The master roadmap owns sequencing and implementation status. This notepad and
|
||||
|
||||
@@ -144,7 +144,7 @@ not require preserving a private completion interface.
|
||||
construction, interpretation, configured validation, and terminal disposition.
|
||||
- Choose the location of engine assembly and focused application helpers without
|
||||
introducing a replacement LLM service layer.
|
||||
- Resolve item 2's deferred filesystem and schema choices and the remaining
|
||||
- Implement item 2's resolved asset/schema design and settle the remaining
|
||||
migration details recorded under items 3–6, including durable compatibility.
|
||||
- Evaluate any remaining upstream gap against an actual application requirement,
|
||||
rather than preserving historical machinery by default.
|
||||
@@ -158,70 +158,40 @@ outcomes as the meaningful Notarius-owned boundaries.
|
||||
|
||||
## 2. Prompt Assets And Schema Assembly
|
||||
|
||||
**Discussion / decision:** Direction agreed: use native PromptKit declarations
|
||||
as the sole prompt assembly mechanism and remove the duplicate application
|
||||
assembly plumbing. Resolution of the custom read-only filesystem and schemas
|
||||
is explicitly deferred; the next topic may clarify those choices.
|
||||
**Discussion / decision:** Agreed and incorporated as resolved U2 in the
|
||||
[master roadmap](promptkit.md#prompt-assets-and-response-schemas-u2-resolved).
|
||||
The filesystem and schema questions initially deferred here are now settled.
|
||||
Implementation is pending; the master roadmap owns the detailed design and checks.
|
||||
|
||||
**Selected direction:** Keep the embedded content tree and load its prompt and
|
||||
profile definitions as native PromptKit configuration in the application
|
||||
collection. Use native filesystem contexts and imports to reference shared
|
||||
content directly. Application composition assembles the embedded and operator
|
||||
sources once; modules select prompt IDs, supply domain inputs, and interpret
|
||||
typed results.
|
||||
Use ordinary embedded assets, native prompt declarations, and native schema
|
||||
loading. Keep the content tree and module ownership of prompt meaning and domain
|
||||
interpretation. Shared files use real native filesystem paths. Remove duplicate
|
||||
asset registration, per-module manifests, synthetic mounts, flattened schema
|
||||
namespaces, and the custom read-only filesystem that served that assembly.
|
||||
|
||||
The spell extractor illustrates the current duplication: its prompt YAML
|
||||
declares ordered messages and a schema, while a separate Go asset manifest
|
||||
lists files, mounts shared fragments, registers filesystems, and hashes them.
|
||||
The selected direction makes native declarations the sole prompt assembly source.
|
||||
Prompt definitions select output contracts by default. Initial and correction
|
||||
prompts can share schema files. Use native per-request replacement only for a
|
||||
demonstrated runtime-dependent contract, without an application wrapper or schema
|
||||
registry. Audita demonstrates ordinary embedded schema loading and per-call
|
||||
contracts; its per-call selection is not required for every Notarius operation.
|
||||
|
||||
Candidates for removal:
|
||||
The default schema source is the embedded asset tree. Operator-selected external
|
||||
schema directories use native whole-root replacement, not an application overlay.
|
||||
Prompt underlays do not implicitly overlay schema files. PromptKit owns private
|
||||
response-schema loading, structural checks, and repair; Notarius keeps typed
|
||||
interpretation, domain validation, and distinct durable artifact schemas/codecs.
|
||||
|
||||
- Prompt and fallback-profile registration plumbing in `AssetRegistry` and
|
||||
individual modules; schema responsibilities remain under review.
|
||||
- `ModulePromptFS`, `PromptAssetManifest`, shared-name lookup tables, synthetic
|
||||
`sharedassets` mounts.
|
||||
- Manual prompt-file enumeration for checkpoint fingerprints, retired under
|
||||
item 5's simpler reuse policy. Native prompt hashes remain available for
|
||||
inspection and diagnostics.
|
||||
Retire response-schema hashes and copied metadata used only by the adapter,
|
||||
checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic
|
||||
snapshots. Neither asset nor schema metadata adds another cache fingerprint.
|
||||
|
||||
Keep prompt prose, ordered messages, cache controls, workload profiles, private
|
||||
response schema documents, and domain interpretation. Durable artifact schemas
|
||||
remain separate. Content ownership does not require runtime asset registration.
|
||||
The spell extractor illustrated the duplication: prompt YAML declared ordered
|
||||
messages and a schema while a Go manifest separately listed, mounted, registered,
|
||||
and hashed assets. Native declarations now supply the selected assembly model.
|
||||
|
||||
Native schema paths resolve against the configured schema root, separately from
|
||||
message-file source contexts. PromptKit v0.23.2 loads schemas during preparation
|
||||
and requires self-contained schema documents. A captured prompt hash alone does
|
||||
not establish schema equality. Existing schema metadata in manifests and
|
||||
checkpoint inputs must be deliberately replaced or retired; its presence does
|
||||
not require preserving the complete wrapper system.
|
||||
|
||||
Deferred decisions:
|
||||
|
||||
- Whether the custom read-only filesystem can be removed after reviewing its
|
||||
remaining responsibilities.
|
||||
- Schema assembly, root/path selection, wrapper removal, and schema metadata.
|
||||
Supplying the embedded filesystem as a native schema root remains a candidate,
|
||||
not a selected design.
|
||||
- Schema metadata for artifact compatibility and inspection. Item 5 settles that
|
||||
cache matching does not require exact schema identity; the CommandKit output
|
||||
decision removes mandatory manifests.
|
||||
|
||||
Implementation questions:
|
||||
|
||||
- Can native sources represent the existing shared fragments and module assets?
|
||||
- Which registry, remounting, and hashing helpers could disappear?
|
||||
- What schema metadata remains useful for inspection and artifact loading under
|
||||
item 5's simpler reuse policy?
|
||||
- How do we preserve message ordering and cache controls while simplifying
|
||||
assembly?
|
||||
|
||||
**Follow-up:** Continue with item 3, then revisit the deferred filesystem and
|
||||
schema decisions with any new clarity. Confirm native source contexts for shared
|
||||
fragments and operator replacements. Apply item 5's reuse policy and resolve
|
||||
remaining metadata with item 6. Verify the resulting definitions through native
|
||||
preparation and fake-provider execution; filesystem remounting tests can retire
|
||||
with the removed implementation. The agreed direction has not yet been implemented.
|
||||
**Follow-up:** Implement the master roadmap's Phase 2 and retire tests of removed
|
||||
mounting/registry machinery. Verify shared paths, operator replacements, native
|
||||
schema sources, and any necessary complete request-contract replacement.
|
||||
|
||||
## 3. Retry And Semantic Correction Ownership
|
||||
|
||||
@@ -335,8 +305,8 @@ Notarius's serialized schema validator currently receives the codec-produced
|
||||
artifact and its artifact schema, which can differ from the model response.
|
||||
Do not remove that boundary merely because both use JSON Schema. Audit actual
|
||||
duplicate response checks individually; artifact validation and Go/domain
|
||||
interpretation are not automatically redundant. Item 2's filesystem and schema
|
||||
assembly decisions remain open.
|
||||
interpretation are not automatically redundant. Item 2's resolved native asset
|
||||
and schema design preserves this distinction.
|
||||
|
||||
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
|
||||
[validation executor](../../internal/framework/pipeline/validation_executor.go),
|
||||
@@ -355,9 +325,8 @@ and the native continuation evidence in item 7.
|
||||
candidate may survive a later failed turn.
|
||||
- Failure handling for non-LLM operations and any justified recovery beyond
|
||||
native execution budgets.
|
||||
- Item 2's deferred filesystem and schema assembly choices, item 5's remaining
|
||||
cache implementation details, and migration to the selected CommandKit output
|
||||
contract.
|
||||
- Item 5's remaining cache implementation details and migration to the selected
|
||||
CommandKit output contract.
|
||||
|
||||
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
|
||||
roadmap. Record the changed architecture in a successor to
|
||||
@@ -825,7 +794,8 @@ Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
|
||||
|
||||
Item 2's native underlay direction has a working application precedent. Audita
|
||||
also offers evidence for ordinary embedded filesystems and per-request schema
|
||||
selection, while our filesystem and schema decisions remain open.
|
||||
selection. U2 now selects ordinary embedded assets and prompt-declared contracts
|
||||
by default, reserving per-request replacement for demonstrated runtime needs.
|
||||
|
||||
The subsequent item 3 discussion selects native continuation and accumulated
|
||||
conversation history. Keeping domain validation local does not require keeping
|
||||
|
||||
@@ -240,8 +240,8 @@ content tree and module ownership of prompt meaning, response requirements, and
|
||||
domain interpretation. Remove prompt/fallback-profile registration in
|
||||
`AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`,
|
||||
shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration
|
||||
for checkpoint fingerprints. Schema-related registry responsibilities remain
|
||||
subject to the explicit schema decision below.
|
||||
for checkpoint fingerprints. The resolved schema design below also removes
|
||||
schema registration and filesystem-flattening machinery.
|
||||
|
||||
Convert maintained resources, examples, and fixtures to native identified lists,
|
||||
`config:` source controls, and sibling inheritance (`extends: {id: parent}`).
|
||||
@@ -268,19 +268,61 @@ for accumulated conversation history; preserving every old byte or retry message
|
||||
is not a migration requirement. Shared content should use native filesystem
|
||||
contexts and imports rather than reconstructed virtual trees.
|
||||
|
||||
PromptKit owns private model-response validation and structural repair. Native
|
||||
schemas resolve against the configured schema root, separately from message-file
|
||||
contexts. The reviewed baseline requires self-contained schemas loaded during
|
||||
preparation.
|
||||
A request override replaces the complete output contract; it must retain the
|
||||
intended schema, validation mode, and repair policy. A prompt hash alone does not
|
||||
establish schema equality.
|
||||
#### Prompt Assets And Response Schemas (U2 Resolved)
|
||||
|
||||
The custom read-only filesystem, schema root/path selection, module-supplied
|
||||
request contracts versus prompt-declared contracts, wrapper removal, and useful
|
||||
schema metadata remain open. An ordinary embedded schema filesystem, as used by
|
||||
Audita, is a candidate rather than a selected requirement. Durable artifact
|
||||
schemas and their validators remain distinct from private response schemas.
|
||||
Use the existing root `assets` package's ordinary `embed.FS` directly for native
|
||||
prompt sources and response schemas, with `fs.Sub` only where useful. Shared
|
||||
message files use their actual paths within the native filesystem context.
|
||||
Remove synthetic mounts, file copying, flattened schema namespaces, and module
|
||||
asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening
|
||||
consumers disappear; no replacement filesystem abstraction is needed.
|
||||
|
||||
Declare response format, validation mode, schema path, and structural-repair
|
||||
budget in native prompt definitions by default. Modules select prompt IDs, supply
|
||||
inputs, and interpret responses. Initial and correction prompts may reference
|
||||
the same schema file. Execution, inspection, and prepared-request preview then
|
||||
use the same declared contract without repeating schema selection in Go.
|
||||
Role-specific prompt contents remain U4, and retry defaults remain U5.
|
||||
|
||||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||||
dependent contract. It replaces the whole configured contract, so supply the
|
||||
intended format, validation mode, schema, and repair policy together. Do not
|
||||
introduce a wrapper or per-call registry solely to reproduce Audita's approach.
|
||||
Operator prompt replacements must still produce responses the selected module
|
||||
can interpret; that module boundary does not require duplicating each prompt's
|
||||
schema selection in Go.
|
||||
|
||||
Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to
|
||||
that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`.
|
||||
They are separate from message-file source contexts: prompt underlays do not
|
||||
implicitly overlay schema files. When an operator configures an external schema
|
||||
directory, select PromptKit's native directory source instead of the embedded
|
||||
root. It replaces the whole schema source and must supply the schemas used by
|
||||
the selected prompts. Do not build an application filesystem overlay to mix
|
||||
embedded and external schemas. Transparent layering would be a separate upstream
|
||||
capability, not a requirement for this migration. Concrete configuration binding
|
||||
is an implementation detail within this selected source policy.
|
||||
|
||||
Keep private response schemas as self-contained JSON documents. PromptKit owns
|
||||
loading, compilation, structural validation, provider structured-output settings,
|
||||
and structural repair. Remove Notarius's response-schema byte registry, copied
|
||||
schema wrappers, defensive-copy layer, digest calculation, and duplicate
|
||||
structural-validation machinery. Native prepared snapshots provide diagnostic
|
||||
schema details; no schema fingerprint is added to the message-content cache key.
|
||||
|
||||
Keep typed decoding, source/reference checks, reconciliation safety, semantic
|
||||
validators, artifact codecs, and durable artifact schemas in Notarius. Response
|
||||
schemas describe model output; artifact schemas describe published and consumed
|
||||
domain artifacts. Preserve identifiers and versions belonging to actual artifact
|
||||
contracts. Retire private schema keys, registration records, hashes, and copied
|
||||
identity/version/name metadata whose only consumers were adapters, checkpoints,
|
||||
or reporting. No independent schema metadata system replaces them.
|
||||
|
||||
Implementation is pending. Verify native preparation of embedded and operator
|
||||
prompt definitions, shared-file paths, schema selection from embedded and external
|
||||
roots, and complete request-contract replacement where actually used. Missing or
|
||||
invalid schemas should follow native preparation behavior. No new schema check
|
||||
requiring synthetic requests is added to offline pipeline preflight.
|
||||
|
||||
### Native Production, Continuation, And Assessment
|
||||
|
||||
@@ -669,7 +711,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
| ID | Remaining choice or dependency | Resolve for |
|
||||
| --- | --- | --- |
|
||||
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U2 | Custom read-only filesystem responsibilities; schema root/path and assembly; prompt-declared versus per-request contracts; removable wrappers and useful artifact-compatibility/inspection metadata. Mandatory manifest metadata is retired. | Phase 2. |
|
||||
| U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. |
|
||||
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
|
||||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||||
@@ -678,6 +719,12 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
| U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. |
|
||||
| U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. |
|
||||
|
||||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||||
native schema-source selection, and removal of duplicate schema/filesystem
|
||||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||||
Implementation remains in Phase 2; role-specific prompts and retry budgets remain
|
||||
U4/U5.
|
||||
|
||||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||||
The former later CommandKit phase is retired. CLI compatibility is a non-goal;
|
||||
@@ -698,8 +745,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
Keep the cache dependency explicit without blocking independent work.
|
||||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||||
forward Audita evidence without adopting its application wrappers.
|
||||
- Preserve U2–U9 with their dependent phases; do not treat resolved CommandKit,
|
||||
debug, cache, or output choices as requirements to redesign.
|
||||
- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas,
|
||||
CommandKit, debug, cache, or output choices as requirements to redesign.
|
||||
|
||||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||||
and outstanding choices are explicit. This phase does not claim that the hash
|
||||
@@ -707,15 +754,17 @@ already exists or that the migration is implemented.
|
||||
|
||||
### Phase 2: Build Native Composition, Sources, And Preflight
|
||||
|
||||
- Resolve U2, update available dependencies, introduce configuration version 6,
|
||||
convert native embedded/companion resources, and establish schema loading.
|
||||
- Implement the resolved U2 design, update available dependencies, introduce
|
||||
configuration version 6, convert native embedded/companion resources, and use
|
||||
native schema loading with embedded/default or operator-selected roots.
|
||||
- Build the small CommandKit application, shared ConfigKit/engine loaders,
|
||||
standard suites, pipeline declarations/native alias, and `internal/preflight`.
|
||||
- Add maintained embedded-only and operator-override examples. Remove duplicated
|
||||
loading and asset assembly as callers migrate.
|
||||
|
||||
**Completion evidence:** runtime/inspection/preflight share intended definitions;
|
||||
source contexts, strict peer conflicts, and replacement work offline. Help,
|
||||
source contexts, shared message paths, prompt-declared output contracts, selected
|
||||
schema roots, strict peer conflicts, and replacement work offline. Help,
|
||||
version, and completion need no configuration. Preflight checks known dependencies
|
||||
without credentials or synthetic requests. `config diff` handles two selections.
|
||||
|
||||
@@ -784,7 +833,8 @@ Release preparation/publication is governed separately by release policy.
|
||||
| Retire or replace | Retained application responsibility |
|
||||
| --- | --- |
|
||||
| `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. |
|
||||
| Prompt/profile `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, manual fingerprint enumeration | Embedded content and module meaning; schema choices under U2. |
|
||||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives | Ordered validators, domain acceptance/rejection/fallback. |
|
||||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||||
@@ -805,7 +855,7 @@ retry tests or freeze complete help text and native debug snapshots.
|
||||
|
||||
| Area | Required evidence |
|
||||
| --- | --- |
|
||||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement and selected schemas work. |
|
||||
| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. |
|
||||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. |
|
||||
|
||||
Reference in New Issue
Block a user