Consolidate migration planning into the master roadmap
This commit is contained in:
@@ -1,30 +1,18 @@
|
||||
# Future Work
|
||||
|
||||
Current Notarius behavior is documented in the canonical README, CLI,
|
||||
configuration, operations, internal, and integration docs. This roadmap records
|
||||
future work only. Items are ordered roughly by current value and specificity,
|
||||
not as committed release dates.
|
||||
|
||||
## Near-Term Validation And LLM Reliability
|
||||
|
||||
PromptKit now owns structural output repair within one completion. Notarius
|
||||
owns stage candidates, validator chains, semantic rejection policy, bounded
|
||||
feedback-aware stage retries, validation provenance, and reusable-state
|
||||
eligibility, and the separation of actionable process warnings from quality
|
||||
diagnostics. The remaining near-term work applies those completed foundations
|
||||
to domain review and empirical evaluation.
|
||||
|
||||
Near-term reliability work should now be selected from the concrete evaluation
|
||||
and extension opportunities below. The retry, validation, and subprocess
|
||||
foundations described above are implemented current behavior.
|
||||
configuration, operations, internal, and integration docs. This backlog records
|
||||
future work outside the [native migration roadmap](promptkit.md), which owns
|
||||
PromptKit/ConfigKit/CommandKit adoption, continuation, admission, cache, output,
|
||||
and debug decisions. Items below are ordered roughly by value and specificity,
|
||||
not as committed release dates or prerequisites for that migration.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
- Evaluate ordinary extraction retries and the completed normalization path
|
||||
against a human-reviewed transcript set before and after adopting the shared
|
||||
PromptKit repair and Notarius validation-retry policies above.
|
||||
- Evaluate extraction, normalization, and correction quality against a
|
||||
human-reviewed transcript set before and after the native migration.
|
||||
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
|
||||
validator, and normalizer development. Treat model-quality review as an
|
||||
iterative human evaluation aid, not a deterministic correctness gate.
|
||||
@@ -46,8 +34,8 @@ The sections below keep broader extensions deferred.
|
||||
### Large-Collection Semantic Reconciliation
|
||||
|
||||
- Evaluate deterministic candidate blocking only after representative registry
|
||||
inputs exceed the active roadmap's bounded single-request limits. Blocking
|
||||
should use cheap, explainable signals to form plausible comparison sets while
|
||||
inputs exceed practical single-request capacity. Blocking should use cheap,
|
||||
explainable signals to form plausible comparison sets while
|
||||
preserving the possibility that a duplicate appears outside a lexical name
|
||||
match.
|
||||
- Define correctness for candidates that appear in more than one block,
|
||||
@@ -61,16 +49,15 @@ The sections below keep broader extensions deferred.
|
||||
|
||||
### Operator-Selected Semantic Policies
|
||||
|
||||
- Consider allowing an operator to select an approved semantic-policy prompt
|
||||
for a typed reconciliation module without replacing the shared protocol,
|
||||
response schema, or deterministic safety rules.
|
||||
- Define the trusted asset source, configuration syntax, compatibility checks,
|
||||
startup validation, provenance, prompt fingerprinting, checkpoint effects,
|
||||
and support boundary before exposing the option.
|
||||
- Prefer selection among registered, typed-policy-compatible prompt assets over
|
||||
arbitrary filesystem prompt paths. Do not add this flexibility until an
|
||||
operator workflow requires it; artifact-family-owned policy remains simpler
|
||||
and safer for the initial implementation.
|
||||
- Consider a dedicated semantic-policy selection option for a typed
|
||||
reconciliation module when a concrete operator workflow needs more than the
|
||||
migration's [native prompt replacement](promptkit.md#configuration-sources-and-prompt-assets).
|
||||
- Use native prompt IDs and sources while retaining typed interpretation and
|
||||
deterministic safety rules. Define any additional configuration and domain
|
||||
compatibility checks from that workflow; do not recreate asset registration,
|
||||
prompt fingerprints, or checkpoint dependency tracking.
|
||||
- Reuse the selected [accepted-artifact cache policy](promptkit.md#accepted-artifact-cache)
|
||||
when evaluating policy changes and the need for explicit refresh.
|
||||
|
||||
### Broader Reconciliation Inputs And Module Selection
|
||||
|
||||
@@ -89,9 +76,8 @@ The sections below keep broader extensions deferred.
|
||||
artifact.
|
||||
- Add production LLM-backed validators only when a concrete review policy
|
||||
benefits from model judgment and deterministic checks are insufficient.
|
||||
- Add validator diagnostics and timing summaries if operators need more detail
|
||||
than the current [durable output bundle](../integrations/json-output.md)
|
||||
provides.
|
||||
- Consider additional validator diagnostic views only if a concrete need remains
|
||||
after the selected [native observations and debug capture](promptkit.md#inspection-and-execution-bookkeeping).
|
||||
- Add validator compatibility metadata if deployments need config-time proof
|
||||
that a validator is suitable for a particular stage, module, or artifact
|
||||
kind.
|
||||
@@ -133,10 +119,11 @@ occurred.
|
||||
|
||||
Define compatibility and migration policy before generated-reference chains
|
||||
must span multiple schema versions or long-lived historical artifacts. The
|
||||
policy should address stable identifier semantics, which schema changes permit
|
||||
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
|
||||
producer or all dependents must be recomputed. Do not add a general migration
|
||||
framework until an actual contract change requires one.
|
||||
policy should address stable identifier semantics, when an older artifact may
|
||||
be decoded or adapted, and when consumers need regenerated artifacts. For cached
|
||||
artifacts, use the [selected reuse policy](promptkit.md#accepted-artifact-cache)
|
||||
and explicit refresh where needed. Do not add a general migration framework
|
||||
until an actual contract change requires one.
|
||||
|
||||
### Artifact-family-oriented physical packaging
|
||||
|
||||
@@ -170,7 +157,6 @@ section only after a concrete workflow, contract, and priority emerge.
|
||||
|
||||
### Workspace And Storage
|
||||
|
||||
- Default-idempotent run behavior with an explicit force override.
|
||||
- Remote workspace storage.
|
||||
- Workspace garbage collection and archival policies.
|
||||
- Cross-machine checkpoint reuse.
|
||||
- Cross-machine sharing of accepted-artifact cache entries.
|
||||
|
||||
Reference in New Issue
Block a user