532 lines
32 KiB
Markdown
532 lines
32 KiB
Markdown
# Configuration Upgrade Review Remediation
|
||
|
||
The earlier PromptKit findings below retain their original scope and status.
|
||
The ConfigKit 1.2.0 review and subsequent focused audits appear below; they
|
||
do not reopen or attribute earlier issues to that upgrade.
|
||
|
||
## Scope And Overall Assessment
|
||
|
||
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
|
||
the accepted [feature roadmap](promptkit.md), and
|
||
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
|
||
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
|
||
PromptKit `/v3` catalogs at v3.20260930.0.
|
||
|
||
The migration is substantially implemented. Native sources, combined operator
|
||
documents and imports, recursive overlays, source-relative prompt content,
|
||
captured engine lifetime, immutable checkpoint identity, and credential-free
|
||
operator inspection are in place. The provider-neutral execution boundary,
|
||
prepared execution, retries, scheduling, and durable artifact contracts remain
|
||
intact. All 16 migrated prompt bodies match the previous definitions after
|
||
removing their old embedded identity fields; prompt fragments and schemas are
|
||
unchanged.
|
||
|
||
Four bounded follow-ups remain below. No production code is changed by this
|
||
review. No additional runtime security restriction, mandatory credential check,
|
||
filesystem permission policy, compatibility shim, or release gate was found.
|
||
Eager final-definition validation and the external keyed format are intentional
|
||
upstream changes, not accidental strengthening. The previously recorded
|
||
ConfigKit remediation was completed before this work set; it is not outstanding.
|
||
|
||
## Finding 1 — Compose Registered Application Assets In One Strict Group
|
||
|
||
Priority: P2. Category: architecture/correctness gap. Status: complete.
|
||
|
||
Completed 2026-11-18: combined application filesystem inputs into one strict
|
||
underlay, retaining their independent contexts. Added conflict/operator-overlay
|
||
coverage; LLM, CLI, and module suites pass, including source-relative content tests.
|
||
|
||
Evidence:
|
||
|
||
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
|
||
registered prompt or fallback-profile filesystem.
|
||
- `internal/framework/llm/asset_registry_sources_test.go`,
|
||
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
|
||
separate registrations yield separate underlays, encoding this divergence.
|
||
- The plan's Sources And Engine Ownership section calls for one application
|
||
underlay whose inputs each retain their own filesystem context;
|
||
`docs/internal/llm.md:86–92` also describes that behavior.
|
||
|
||
Separate underlays have recursive overlay semantics. Conflicting application
|
||
definitions in different registrations therefore silently depend on module
|
||
registration order instead of receiving native strict-composition diagnostics.
|
||
An offline probe registered two fallback files with the same profile key and
|
||
different models. The current inspector accepted them and selected the second
|
||
model. The exact same filesystem inputs placed in one native underlay correctly
|
||
failed composition. Distinct filesystem contexts do not require separate layers.
|
||
|
||
Required changes:
|
||
|
||
- Gather the native `FilesystemInputs` from every application prompt and
|
||
fallback-profile registration into one application underlay. Preserve each
|
||
input's existing label, filesystem, root, and disabled-import setting. Keep
|
||
schemas on their separate runtime option and omit an empty application group.
|
||
- Keep the operator and explicit local-backend contributions as later groups.
|
||
Preserve their accepted recursive overrides and source precedence. Do not
|
||
add an application-specific duplicate-key detector or flatten filesystem
|
||
contexts into an artificial shared root.
|
||
- Replace the incidental group-count test with behavioral coverage: conflicting
|
||
application contributions fail through native strict composition; distinct
|
||
declarations retain source-relative content; an operator overlay over the
|
||
resulting application base still succeeds. Reuse existing origin coverage.
|
||
|
||
Acceptance: application conflicts are not silently resolved by registration
|
||
order; source-relative imports/content and valid later operator overlays remain
|
||
unchanged. Runtime, preflight, and reports use the corrected shared construction.
|
||
|
||
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
|
||
|
||
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
|
||
|
||
Priority: P3. Category: CLI correctness. Status: complete.
|
||
|
||
Completed 2026-11-18: both scopes validate canonical pointer syntax before
|
||
loading configuration. Missing-file regression and CLI/LLM suites pass.
|
||
|
||
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
|
||
Notarius configuration and constructs the inspection engine before validating
|
||
the pointer; the default Notarius scope parses the pointer first.
|
||
|
||
Reproduction, using the native temporary build and a nonexistent config path:
|
||
|
||
```sh
|
||
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
|
||
```
|
||
|
||
The current PromptKit command returns exit 1 with a config-file error. The same
|
||
malformed pointer returns exit 2 with a valid config, and the Notarius scope
|
||
returns exit 2 even with the missing config. A bad PromptKit definition or
|
||
missing content file can likewise mask malformed command syntax. This differs
|
||
from the selected syntax/runtime distinction in the plan and CLI reference.
|
||
|
||
Required changes:
|
||
|
||
- Validate canonical JSON Pointer syntax once, before either scope loads config
|
||
or constructs an engine. Reuse the existing parser; retain defensive parsing
|
||
inside the adapter for direct callers.
|
||
- Keep the empty root pointer, escaped literal keys, historical-only paths,
|
||
unknown-path exit 1, and writer-error behavior unchanged.
|
||
- Extend CLI coverage with a malformed pointer and unavailable configuration;
|
||
verify exit 2, no stdout, and no configuration/provider work. No broader
|
||
argument-parser refactor is needed.
|
||
|
||
Acceptance: malformed pointers are syntax errors independent of configuration
|
||
availability for both scopes; valid selectors still load and inspect normally.
|
||
|
||
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
|
||
|
||
## Finding 3 — Correct Lifecycle And Fallback Documentation
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: complete.
|
||
|
||
Completed 2026-11-18: corrected construction/preparation timing, fallback overlay
|
||
semantics, and operator/local-backend alternatives in comments and canonical docs.
|
||
Checked against current source construction and existing passing inspection tests;
|
||
affected links and whitespace checks pass. No runtime behavior changed.
|
||
|
||
Evidence:
|
||
|
||
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
|
||
at engine construction. Schemas remain preparation-time resources supplied
|
||
through `WithSchemaFS`, not captured configuration definitions.
|
||
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
|
||
as occurring without loading a prompt. Construction now loads and validates
|
||
effective prompt definitions and captures message files/static templates;
|
||
it does not prepare a request or generate output.
|
||
- The comment on `RegisterFallbackProfileFS` in
|
||
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
|
||
are used only when the operator source lacks the ID. Under recursive overlays,
|
||
their omitted fields remain active beneath a matching operator entry.
|
||
|
||
Required changes:
|
||
|
||
- Distinguish constructor-time definition/message capture from preparation-time
|
||
schema loading and request-variable/artifact resolution. State accurately
|
||
that preflight needs no transcript, credential value, or provider call; do
|
||
not promise it performs no prompt-content file reads.
|
||
- Describe fallback registration as a lower-precedence application contribution
|
||
rather than an all-or-nothing lookup fallback.
|
||
- While editing this text, clarify that an operator `backends.local` declaration
|
||
can supply the backend as well as Notarius's optional `local_backend` object.
|
||
The latter is a convenience contribution, not a required second declaration.
|
||
Link canonical source rules rather than duplicating them.
|
||
- Change documentation/comments only unless verification exposes a real runtime
|
||
defect. Do not move schema loading earlier to make the stale prose true.
|
||
|
||
Acceptance: public and internal guidance consistently explains actual lifecycle,
|
||
native layering, and operator backend configuration without inventing a stronger
|
||
preflight guarantee or an additional required config field.
|
||
|
||
Validation: inspect current constructors/prepared execution and upstream v0.12
|
||
contracts; review affected local links; run `git diff --check`. Reuse unchanged
|
||
behavioral checks rather than adding prose-snapshot tests.
|
||
|
||
## Finding 4 — Preserve ADR-0010's Historical Decision Body
|
||
|
||
Priority: P3. Category: documentation-policy compliance. Status: complete.
|
||
|
||
Completed 2026-11-18: restored the original body from f5b066b, retaining only
|
||
the approved status/supersession change. Baseline diff and link checks pass.
|
||
|
||
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
|
||
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
|
||
and `:47–50`, as well as adding the appropriate status/supersession link.
|
||
`docs/policy/documentation.md` requires accepted decision content to remain
|
||
immutable; Stage 14 specified a concise status link rather than rewriting the
|
||
historical body.
|
||
|
||
The replacement correctly acknowledges the old behavior, but the historical
|
||
record should retain its original wording. ADR-0019 already owns the new native
|
||
overlay decision and accurately limits which part of ADR-0010 it supersedes.
|
||
|
||
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
|
||
keeping its new partial-supersession status and ADR-0019 link. Leave workload
|
||
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
|
||
in the canonical current references, not retrofitted into the older decision.
|
||
|
||
Acceptance: the historical body matches the accepted original, the changed
|
||
clause's supersession remains discoverable, and no runtime behavior changes.
|
||
|
||
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
|
||
verify the ADR link, and run `git diff --check`.
|
||
|
||
## Review Evidence And Validation
|
||
|
||
- `GOWORK=off go test ./...` passed.
|
||
- `GOWORK=off go vet ./...` passed.
|
||
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
|
||
- `GOWORK=off go mod tidy -diff` produced no changes.
|
||
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
|
||
- A native build outside the repository succeeded. Credential-free profile and
|
||
explanation commands and complete/development pipeline validation succeeded.
|
||
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
|
||
are unchanged after removing old embedded IDs/versions. Prompt fragments,
|
||
schemas, and the embedded D&D execution settings are unchanged.
|
||
- An offline Go overlay probe, stored entirely outside the repository, confirmed
|
||
the application-group conflict behavior in Finding 1. CLI invocations
|
||
reproduced Finding 2 without generation or provider calls.
|
||
- The implementation diff passed whitespace checks. No production inference
|
||
or production model-quality evaluation was performed.
|
||
|
||
Recommended remediation is limited to these findings, not another configuration
|
||
system, new security policy, or unrelated pipeline refactoring.
|
||
|
||
## ConfigKit 1.2.0 Review
|
||
|
||
Reviewed on 2026-11-17 against implementation revision `c9744ad`, compared with
|
||
planning revision `79088a3` and the ConfigKit feature roadmap and implementation
|
||
plan. The subsequent roadmap-only clarification in `d88a01c` does not change
|
||
the reviewed runtime. ConfigKit v1.2.0 resolves to
|
||
`c1c7e87f2cdf77a0cbf3327831ac05578e756b9e`.
|
||
|
||
The production architecture matches the target. Application configuration uses
|
||
new-path ConfigKit v1.2.0; the unchanged PromptKit v0.12.0 adapter retains
|
||
old-path ConfigKit v0.5.0. Both use YAML v4.0.0-rc.6 under Go 1.27.0. The CLI
|
||
describes one exclusive file or ordered native search source, while ConfigKit
|
||
owns acquisition, imports, collections, inheritance, decoding, diagnostics, and
|
||
inspection. Notarius supplies domain defaults, recorded normalization,
|
||
validation, detached runtime conversion, and catalog-dependent resolution.
|
||
|
||
No new production correctness defect or unwarranted restriction was found.
|
||
The changed controls, optional import defaults, post-decode CLI assignments,
|
||
step identity/upsert, search behavior, and higher Go minimum are approved scope.
|
||
The remaining gaps are two operator-documentation inaccuracies from Stage 9,
|
||
also missed by Stage 11's final documentation check. Production fixes are not
|
||
required for either finding.
|
||
|
||
### Finding CK-1 — Distinguish Mapping And Record Inheritance Syntax
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: complete.
|
||
|
||
Completed 2026-11-18: documented both reference forms and their lookup scopes,
|
||
linking to the pinned native examples. Checked against v1.2.0's contract and
|
||
existing inheritance coverage; no non-executable example was added.
|
||
|
||
Evidence: `docs/config.md:143–149` introduces both mapping and identified-record
|
||
inheritance, then says a directive names an immediate sibling in the same
|
||
mapping. That describes mapping inheritance only. For an identified step record,
|
||
the parent belongs to the same sequence and the reference must be an identity
|
||
mapping, such as `extends: {id: parent}`. The reference never shows that form.
|
||
|
||
An offline CLI probe with a parent step providing references and a child step
|
||
providing an artifact lane confirmed that `extends: parent` fails with
|
||
`inheritance.invalid`, while changing only the directive to
|
||
`extends: {id: parent}` loads successfully and preserves the inherited reference.
|
||
`TestIdentifiedStepRecordInheritance` already exercises the correct form in
|
||
`internal/core/config/collection_composition_test.go:182–208`.
|
||
|
||
Required changes:
|
||
|
||
- Explain string-valued mapping-sibling inheritance and identity-mapping
|
||
sequence-record inheritance separately, including their distinct lookup scopes.
|
||
- Add a minimal step-record example or a direct link to the pinned upstream
|
||
syntax example. Preserve the existing rules for null cancellation, validation,
|
||
and whole-record upsert; no new application inheritance machinery is needed.
|
||
|
||
Acceptance: an operator can determine the correct syntax for both a pipeline
|
||
mapping and an identified record from the canonical reference. Any example
|
||
presented as a runnable pipeline must pass pipeline resolution, not just loading;
|
||
see CK-3 for step-record limitations. Reuse behavioral coverage rather than adding
|
||
a documentation snapshot or changing runtime behavior to fit the prose.
|
||
|
||
### Finding CK-2 — Describe Dependency-Ready Step Assembly Order
|
||
|
||
Priority: P3. Category: documentation accuracy. Status: complete.
|
||
|
||
Completed 2026-11-18: replaced the source-order claim with native dependency-ready
|
||
assembly, linked the pinned contract, and explained the complete example's ordering.
|
||
Verified against the pinned scheduler contract and unchanged example assertions.
|
||
|
||
Evidence: `docs/config.md:167–170` says distinct imported step IDs combine
|
||
"in source order." ConfigKit distinguishes acquisition/import declaration order
|
||
from dependency-ready assembly order. Ready contributions are selected by their
|
||
stable source keys, so independent branches can interleave. The feature roadmap
|
||
and Stage 8 correctly rely on dependency-ready assembly rather than a Notarius
|
||
append pass.
|
||
|
||
An offline CLI probe declared imports in the order `z-first.yml`, then
|
||
`a-second.yml`, each contributing a different step and lane. Effective output
|
||
placed the step from `a-second.yml` first, then the step from `z-first.yml`.
|
||
Thus declaration order does not determine the resulting step list. This matters
|
||
because Notarius executes steps in the resolved list order and validates their
|
||
generated-reference dependencies.
|
||
|
||
Required changes:
|
||
|
||
- Replace the source-order claim with the native dependency-ready assembly rule
|
||
and link to ConfigKit v1.2.0's pinned composition contract for its full ordering
|
||
semantics. Do not promise declaration order or duplicate the upstream scheduler.
|
||
- Explain briefly why the maintained complete example's imported describe-session
|
||
step precedes the importing body's extract-events step: the imported dependency
|
||
is assembled before its importer. Keep the canonical operator rule in
|
||
`docs/config.md`; other documentation should link rather than redefine it.
|
||
|
||
Acceptance: the reference does not imply that reordering independent import
|
||
declarations necessarily reorders steps. The complete example's actual step
|
||
order and generated-reference handoffs remain unchanged. No custom sorting,
|
||
merging, or scheduler changes belong in this remediation.
|
||
|
||
### Stage-By-Stage Review Evidence
|
||
|
||
| Stage | Review result |
|
||
| --- | --- |
|
||
| 1 — Dependencies/toolchain | The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff. |
|
||
| 2 — API/handoff | Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values. |
|
||
| 3 — Normalization/origins | Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass. |
|
||
| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, and source-relative references. Identified-record inheritance is checked only at loading/configuration validation; CK-3 records the missing runtime-consumer check. |
|
||
| 5 — Fixtures | Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes. |
|
||
| 6 — CLI discovery | The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed. |
|
||
| 7 — Discovery/failures | Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files. |
|
||
| 8 — Examples | Minimal, complete, and development pipelines validate without inference; the supporting fragment preserves module choices, step order, and handoffs. |
|
||
| 9 — Operator docs | Search, controls, path bases, CLI timing, and toolchain guidance are updated. CK-1 and CK-2 remain. |
|
||
| 10 — Internal/integration docs and notices | The mixed dependency boundary and pinned ConfigKit attribution match the selected graph. Accepted ADRs, policy, and project license/grant were not modified by the sprint. |
|
||
| 11 — Final verification | Full tests, vet, tidy, native build, dependency inspection, and provider-free validation/effective/explanation commands pass. The final documentation review missed CK-1 and CK-2. |
|
||
|
||
### ConfigKit Review Validation
|
||
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./...` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go vet ./...` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test -race ./internal/core/config ./internal/cli` passed.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go mod tidy -diff` produced no changes.
|
||
- A native build in a temporary directory succeeded. Minimal, complete, and
|
||
development configuration validation, plus application and PromptKit effective
|
||
and explanation reports, succeeded without provider calls.
|
||
- Selected module inspection confirmed the four approved ConfigKit, PromptKit,
|
||
and YAML versions. Pinned local upstream documentation and notices were checked.
|
||
- Temporary fixtures outside the repository reproduced both documentation gaps
|
||
and verified combined identifier/reference-path normalization. No production
|
||
pipeline, credential lookup, or live model-quality evaluation was performed.
|
||
- The review document passed `git diff --check`; production code and current
|
||
operator documentation were not changed by this review.
|
||
|
||
Keep the earlier PromptKit findings separate from these two documentation fixes.
|
||
The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope.
|
||
|
||
## ConfigKit Integration Audit
|
||
|
||
Reviewed on 2026-11-17 against `6f0a75e`, with particular attention to how the
|
||
11 implementation stages interact with the existing pipeline and PromptKit
|
||
consumers. The implementation is unchanged from `c9744ad`; the intervening
|
||
commits changed roadmaps only. CK-1 and CK-2 remain open.
|
||
|
||
The dependency/toolchain boundary, detached configuration conversion, path
|
||
normalization, CLI overrides, native discovery, selected-file reporting, and
|
||
old-format PromptKit source handoff are coherent. The maintained complete
|
||
example still reaches the existing multi-step runner and publishes its generated
|
||
registry/occurrence handoffs through an injected external client. Session,
|
||
checkpoint, warning, and output consumers continue to receive resolved domain
|
||
values, not ConfigKit source graphs. No new production correctness regression
|
||
was identified. One additional test/documentation integration gap remains.
|
||
|
||
### Finding CK-3 — Verify Identified-Record Inheritance At The Runtime Boundary
|
||
|
||
Priority: P3. Category: integration coverage and support-scope clarity. Status: open.
|
||
Affected stages: 4, 9, and 11.
|
||
|
||
`TestIdentifiedStepRecordInheritance` in
|
||
`internal/core/config/collection_composition_test.go:182–208` supplies a parent
|
||
step containing only references and a child step containing an artifact lane.
|
||
It checks `Load` and `Config.Validate`, but never calls `Config.Resolve` with a
|
||
catalog. The existing runner's resolver requires every explicit step to have at
|
||
least one artifact lane (`internal/framework/pipeline/profile.go`,
|
||
`ResolvePipeline`). The parent is an ordinary executable step, not an excluded
|
||
template. The fixture therefore cannot be used as a resolved pipeline, despite
|
||
the positive test and the reference's unqualified mention of step-record
|
||
inheritance.
|
||
|
||
Giving the parent a real artifact lane is not a simple repair to the fixture:
|
||
native inheritance recursively copies that named lane into the child, while
|
||
Notarius requires lane identifiers to be unique across steps. Adding a new
|
||
child-specific lane does not remove the inherited one. Pipeline-level inheritance
|
||
with step upsert is a different operation and remains supported and tested.
|
||
|
||
Offline reproduction through the current binary:
|
||
|
||
- A production-module equivalent of the test fixture loads for effective
|
||
inspection, but `config validate --pipeline session` fails because the parent
|
||
step must declare an artifact lane.
|
||
- A parent with `parent-spells` and a child declaring `child-spells` fails domain
|
||
validation because inherited `parent-spells` occurs in both steps.
|
||
- A native identity declaration on an extraction validator list, with
|
||
`generic/valid_json_schema` inheriting from `generic/valid_json` through
|
||
`extends: {module: generic/valid_json}`, passes production pipeline validation.
|
||
Generic identified-record inheritance is therefore usable; the gap is the
|
||
selected positive step fixture and its implied runtime applicability.
|
||
|
||
Required changes:
|
||
|
||
- Replace or reframe the positive step fixture. Cover a genuinely executable
|
||
identified-record use case in a compatible collection and verify it through
|
||
`Config.Resolve` with the appropriate catalog or the equivalent public CLI
|
||
validation path. Assert a meaningful inherited value, not only successful load.
|
||
- Make the step-specific constraints clear alongside CK-1's syntax correction.
|
||
Do not present the reference-only parent fixture as a runnable step template.
|
||
Preserve the distinction between pipeline inheritance/upsert and inheritance
|
||
between records of one final sequence.
|
||
- Preserve the existing runtime constraints. This finding does not authorize
|
||
template-only steps, implicit parent removal, lane renaming, or weaker lane-ID
|
||
uniqueness. A separate product decision would be needed for such features.
|
||
|
||
Acceptance: positive inheritance coverage reaches the actual configuration
|
||
consumer, and operator guidance distinguishes native composition from executable
|
||
pipeline validity. The complete D&D example, generated-reference validation, and
|
||
ordinary pipeline inheritance continue to behave unchanged. No production fix
|
||
is indicated by the current evidence.
|
||
|
||
### Integration Audit Validation
|
||
|
||
- Reviewed the staged changes against configuration conversion, pipeline
|
||
resolution/reference materialization, PromptKit construction/inspection,
|
||
checkpoint identity, and CLI debug/output consumers.
|
||
- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline ./internal/modules/integration`
|
||
passed. This includes the maintained complete-example run, generated-reference
|
||
handoffs, command overrides, selected-path debug provenance, and profile reports.
|
||
- Temporary CLI fixtures outside the repository reproduced both step-inheritance
|
||
failures and verified the compatible validator-record alternative. An additional
|
||
author-identified validator list with whitespace-normalized module names also
|
||
passed pipeline validation.
|
||
- The full-suite, vet, race, tidy, and build results from the preceding review
|
||
remain applicable to the unchanged implementation. No live provider call or
|
||
production-quality evaluation was performed.
|
||
|
||
## ConfigKit Algorithmic Audit
|
||
|
||
Reviewed on 2026-11-18 against `b396581`. Production code remains identical to
|
||
`c9744ad`. This pass examines the 11 stages for algorithmic correctness and
|
||
avoidable scaling costs; it leaves the explicitly selected native composition,
|
||
inheritance, discovery, and ownership algorithms in place. CK-1 through CK-3
|
||
retain their earlier scope and status.
|
||
|
||
### Finding CK-4 — Memoize Source Directories During Path Normalization
|
||
|
||
Priority: P3. Category: efficiency/scalability opportunity. Status: open.
|
||
Affected stages: 2–3; exposed by adoption of richer ConfigKit source metadata.
|
||
|
||
`internal/core/config/document_paths.go:144–176`, `documentPathBase`, calls
|
||
`resolution.Source(sourceID)` for each file-authored relative path. Only the
|
||
source's `Path` is needed to calculate its directory. The same source is queried
|
||
again for every reference or destination it supplies.
|
||
|
||
In ConfigKit v1.2.0, `Source` returns a detached record that includes
|
||
`EmptyImportSelections`: one record for each completed optional import declaration
|
||
that selected no source. The pinned implementation clones these records and
|
||
their paths on each query:
|
||
|
||
- `lifecycle.go:287–290`: public resolution-source lookup and conversion;
|
||
- `internal/engine/load_metadata.go:47–52`: indexed lookup followed by a detached copy;
|
||
- `internal/engine/records.go:333–345`: copying of the empty-selection evidence;
|
||
- `conversion.go:45–66`: conversion of that evidence to the public record.
|
||
|
||
The older v0.5.0 source record did not contain this variable-sized evidence.
|
||
The per-field lookup pattern predates the upgrade, but its snapshot cost now
|
||
depends on metadata unrelated to the particular path being normalized. The
|
||
source-ID lookup itself is indexed; the repeated copying is the avoidable work.
|
||
|
||
Complexity, disregarding path-string lengths:
|
||
|
||
- Let `R` be the number of relative fields supplied by one source and `E` its
|
||
number of empty optional-import selections. Repeated source lookup contributes
|
||
`O(R * (1 + E))` time and cumulative allocation volume. With `R` and `E` growing
|
||
together, this component is quadratic in their combined count.
|
||
- A load-local cache of source ID to its directory/lookup result reduces this
|
||
component to `O(R + 1 + E)` for one source. Across visited sources, the bound is
|
||
`O(R + U + sum(E_i))`, with `O(U)` small cached entries for `U` source IDs.
|
||
- The cache should retain only the needed path-derived result, not every full
|
||
source snapshot. One-time snapshotting for inspection and main-source selection
|
||
can remain unchanged.
|
||
|
||
Recommended refactoring:
|
||
|
||
- Memoize the source-path/directory lookup within `documentLoadState`, scoped to
|
||
one load and its resolution callback. Keep per-field `EntryInfo` queries so
|
||
each surviving declaration still chooses its own source correctly.
|
||
- Preserve CLI-invocation-relative, default-main-relative, imported/inherited,
|
||
missing-source/non-file fallback, absolute-path, and empty-cache behavior.
|
||
Memoization should reuse the native immutable source evidence without changing
|
||
source acquisition or application path policy.
|
||
- Reuse the existing path-origin and ownership tests. Verify the lookup is shared
|
||
across paths from one source and remains distinct across sources; no elapsed-time
|
||
assertion, real-world benchmark, new input limit, or release gate is warranted.
|
||
|
||
Acceptance: paths from the same source no longer repeatedly clone its unrelated
|
||
import-selection evidence. Runtime paths, inspection results, and load isolation
|
||
remain unchanged. This is a bounded optimization opportunity, not a correctness
|
||
failure or a claim that ordinary inference runs will become materially faster.
|
||
For sources with no empty selections, the current lookup cost is already small.
|
||
|
||
### Stage-By-Stage Algorithmic Assessment
|
||
|
||
| Stage | Assessment |
|
||
| --- | --- |
|
||
| 1 — Dependencies/toolchain | The mixed released graph is an approved choice; no new Notarius runtime algorithm was introduced here. |
|
||
| 2 — API/handoff | LoadSource is a single delegation. MainSourcePath makes one source-metadata snapshot and scan, linear in that snapshot's size, rather than repeating discovery. CK-4 concerns repeated callback lookups, not this one-time accessor. |
|
||
| 3 — Normalization/origins | Local passes visit each contained pipeline, step, lane, validator, and reference a bounded number of times. Deterministic key sorting costs `O(k log k)` per map. CK-4 identifies a hidden multiplicative snapshot cost inside those traversals. |
|
||
| 4 — Collections | Identity registration delegates to the selected native algorithms. No application-side step matcher, concatenation loop, or quadratic merge was added. |
|
||
| 5 — Fixtures | Mechanical fixture migration introduces no production algorithm. Previously recorded correctness/coverage findings remain separate. |
|
||
| 6 — CLI discovery | Source selection has a fixed number of local branches and native preset descriptions; manual probes were removed. Selected-source reporting is performed once. |
|
||
| 7 — Discovery/inspection coverage | The new tests exercise bounded cases; reports delegate to retained inspection. No extra load or source search was added for provenance. |
|
||
| 8 — Examples | The extra fragment uses native dependency ordering and does not add application composition machinery. |
|
||
| 9 — Operator docs | No runtime algorithm introduced. CK-1 and CK-2 already cover the relevant prose inaccuracies. |
|
||
| 10 — Internals/notices | No runtime algorithm introduced or new dependency-graph translation added. |
|
||
| 11 — Final verification | No production algorithm introduced; existing tests support correctness but do not establish complexity bounds. |
|
||
|
||
### Algorithmic Audit Evidence And Limits
|
||
|
||
- Inspected the production diff and traced traversal, original-path remapping,
|
||
recording, source lookup, and CLI handoff helpers through the code graph.
|
||
Nested collection loops traverse separate children rather than repeatedly
|
||
scanning one global collection. Original-path remapping is bypassed during the
|
||
pre-move path pass; its later worker-presence lookup has a fixed schema path.
|
||
- Inspected the exact ConfigKit v1.2.0 source for snapshot ownership costs. Its
|
||
record aggregation uses indexed reason/dependency sets, so repeated ancestor
|
||
recording was not misclassified as a quadratic scan.
|
||
- An offline temporary Go-overlay test verified that a single declaring-source
|
||
snapshot contains four empty optional-import records while multiple relative
|
||
reference paths normalize correctly. The probe passed; its files remain outside
|
||
the repository. No production code or repository tests were modified.
|
||
- Complexity conclusions come from source/data-flow analysis. No production
|
||
inference, timing benchmark, or real-world performance measurement was used.
|
||
Existing validation results remain applicable to the unchanged implementation.
|
||
|
||
No additional ordinary correctness bug was identified in this pass.
|