Consolidate migration planning into the master roadmap

This commit is contained in:
2026-11-20 13:06:52 +00:00
parent f883316d7a
commit 2d69c26813
6 changed files with 124 additions and 2166 deletions

View File

@@ -1,30 +1,18 @@
# Future Work
Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity,
not as committed release dates.
## Near-Term Validation And LLM Reliability
PromptKit now owns structural output repair within one completion. Notarius
owns stage candidates, validator chains, semantic rejection policy, bounded
feedback-aware stage retries, validation provenance, and reusable-state
eligibility, and the separation of actionable process warnings from quality
diagnostics. The remaining near-term work applies those completed foundations
to domain review and empirical evaluation.
Near-term reliability work should now be selected from the concrete evaluation
and extension opportunities below. The retry, validation, and subprocess
foundations described above are implemented current behavior.
configuration, operations, internal, and integration docs. This backlog records
future work outside the [native migration roadmap](promptkit.md), which owns
PromptKit/ConfigKit/CommandKit adoption, continuation, admission, cache, output,
and debug decisions. Items below are ordered roughly by value and specificity,
not as committed release dates or prerequisites for that migration.
## Near-Term D&D Pipeline
### Evaluate Spell Extraction And Normalization
- Evaluate ordinary extraction retries and the completed normalization path
against a human-reviewed transcript set before and after adopting the shared
PromptKit repair and Notarius validation-retry policies above.
- Evaluate extraction, normalization, and correction quality against a
human-reviewed transcript set before and after the native migration.
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
validator, and normalizer development. Treat model-quality review as an
iterative human evaluation aid, not a deterministic correctness gate.
@@ -46,8 +34,8 @@ The sections below keep broader extensions deferred.
### Large-Collection Semantic Reconciliation
- Evaluate deterministic candidate blocking only after representative registry
inputs exceed the active roadmap's bounded single-request limits. Blocking
should use cheap, explainable signals to form plausible comparison sets while
inputs exceed practical single-request capacity. Blocking should use cheap,
explainable signals to form plausible comparison sets while
preserving the possibility that a duplicate appears outside a lexical name
match.
- Define correctness for candidates that appear in more than one block,
@@ -61,16 +49,15 @@ The sections below keep broader extensions deferred.
### Operator-Selected Semantic Policies
- Consider allowing an operator to select an approved semantic-policy prompt
for a typed reconciliation module without replacing the shared protocol,
response schema, or deterministic safety rules.
- Define the trusted asset source, configuration syntax, compatibility checks,
startup validation, provenance, prompt fingerprinting, checkpoint effects,
and support boundary before exposing the option.
- Prefer selection among registered, typed-policy-compatible prompt assets over
arbitrary filesystem prompt paths. Do not add this flexibility until an
operator workflow requires it; artifact-family-owned policy remains simpler
and safer for the initial implementation.
- Consider a dedicated semantic-policy selection option for a typed
reconciliation module when a concrete operator workflow needs more than the
migration's [native prompt replacement](promptkit.md#configuration-sources-and-prompt-assets).
- Use native prompt IDs and sources while retaining typed interpretation and
deterministic safety rules. Define any additional configuration and domain
compatibility checks from that workflow; do not recreate asset registration,
prompt fingerprints, or checkpoint dependency tracking.
- Reuse the selected [accepted-artifact cache policy](promptkit.md#accepted-artifact-cache)
when evaluating policy changes and the need for explicit refresh.
### Broader Reconciliation Inputs And Module Selection
@@ -89,9 +76,8 @@ The sections below keep broader extensions deferred.
artifact.
- Add production LLM-backed validators only when a concrete review policy
benefits from model judgment and deterministic checks are insufficient.
- Add validator diagnostics and timing summaries if operators need more detail
than the current [durable output bundle](../integrations/json-output.md)
provides.
- Consider additional validator diagnostic views only if a concrete need remains
after the selected [native observations and debug capture](promptkit.md#inspection-and-execution-bookkeeping).
- Add validator compatibility metadata if deployments need config-time proof
that a validator is suitable for a particular stage, module, or artifact
kind.
@@ -133,10 +119,11 @@ occurred.
Define compatibility and migration policy before generated-reference chains
must span multiple schema versions or long-lived historical artifacts. The
policy should address stable identifier semantics, which schema changes permit
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
producer or all dependents must be recomputed. Do not add a general migration
framework until an actual contract change requires one.
policy should address stable identifier semantics, when an older artifact may
be decoded or adapted, and when consumers need regenerated artifacts. For cached
artifacts, use the [selected reuse policy](promptkit.md#accepted-artifact-cache)
and explicit refresh where needed. Do not add a general migration framework
until an actual contract change requires one.
### Artifact-family-oriented physical packaging
@@ -170,7 +157,6 @@ section only after a concrete workflow, contract, and priority emerge.
### Workspace And Storage
- Default-idempotent run behavior with an explicit force override.
- Remote workspace storage.
- Workspace garbage collection and archival policies.
- Cross-machine checkpoint reuse.
- Cross-machine sharing of accepted-artifact cache entries.