scriptorium/docs/roadmap/implementation.md

# Built-In Profiles And Library API Implementation Plan

This plan implements the target states in:

- `docs/roadmap/builtins.md`
- `docs/roadmap/library.md`

Audience: LLM coding agents implementing the work in order. Review and follow `docs/policy/architecture.md`, `docs/policy/development.md`, and `docs/policy/documentation.md` before changing code.

## Global Constraints

- Implement built-in profiles before the public library production upgrades.
- Keep orchestration in `internal/usecase`; adapters and the public package should translate inputs and wire components.
- Keep `internal/*` packages internal. Public package types must remain facade types.
- Do not add a mutable global profile registry.
- Do not add a credential resolver or secret-manager abstraction.
- Do not accept raw API keys in YAML/JSON config, profile files, prompt files, CLI flags, or HTTP request bodies.
- Do not emit raw API keys in prepared output, run results, logs, or examples.
- Prefer standard library APIs. Do not add dependencies unless a later implementation prompt explicitly approves one.
- Keep each stage passing `go test ./...` before moving to the next stage.

## Stage 1: Profile Repository Foundations

Goal: add reusable profile repository primitives that support built-ins without changing runner behavior.

### Implementation Steps

1. Add an `fs.FS`-backed profile repository in `internal/profile`.
   - Constructor shape should be similar to `NewFSRepository(fsys fs.FS, root string) Repository`.
   - It must scan YAML files recursively below `root`.
   - It must use the same strict YAML decoding, profile validation, raw `api_key` rejection, and duplicate-ID behavior as the existing filesystem repository.
   - It must return the existing profile package sentinel errors where applicable.

2. Refactor shared profile-loading behavior.
   - Avoid duplicating validation and metadata logic between filesystem and `fs.FS` repositories.
   - The existing `NewFilesystemRepository(dir)` API should remain available.
   - It may either delegate to the `fs.FS` repository through `os.DirFS` or share unexported loader helpers.

3. Add an overlay profile repository in `internal/profile`.
   - Constructor shape should be similar to `NewOverlayRepository(primary, fallback Repository) Repository`.
   - Lookup must return the primary result when primary succeeds.
   - Lookup must fall back only when `errors.Is(err, profile.ErrProfileNotFound)` for the primary.
   - Lookup must return primary load/validation errors directly and must not fall back after those errors.
   - Nil repository inputs should be handled deliberately. Prefer treating nil primary as "no primary" and requiring a non-nil fallback for built-in-only operation.

### Tests

Add focused tests under `internal/profile`.

Required coverage:

- `NewFSRepository` loads valid profiles from nested directories.
- `NewFSRepository` rejects unknown YAML fields.
- `NewFSRepository` rejects raw `api_key` in the selected profile.
- `NewFSRepository` ignores raw `api_key` in non-selected profiles, matching current filesystem behavior.
- `NewFSRepository` rejects duplicate IDs within one source.
- `NewFilesystemRepository` still satisfies all existing repository tests.
- `NewOverlayRepository` returns primary matches before fallback matches.
- `NewOverlayRepository` falls back on primary not found.
- `NewOverlayRepository` does not fall back after a primary invalid YAML/profile/raw-key error.
- `NewOverlayRepository` returns not found when both sources miss.

### Verification

Run:

```bash
go test ./internal/profile
go test ./...
```

## Stage 2: Built-In Profile Assets And Wiring

Goal: compile built-in profiles into Scriptorium and make profile lookup use user-over-built-in precedence.

### Implementation Steps

1. Add an `internal/profile/builtin` package.
   - Store built-in profile YAML files in a stable asset directory under that package.
   - Use Go `embed` to compile those files into the binary/package.
   - Expose a constructor such as `builtin.NewRepository() profile.Repository`.
   - The repository should use the `fs.FS` profile repository from Stage 1.

2. Add built-in profile validation tests.
   - Tests should load every built-in profile through the real profile loader.
   - Tests should fail if the built-in profile set contains duplicate IDs.
   - Tests should fail if any built-in profile contains raw `api_key`.

3. Add built-in profiles.
   - Use `docs/roadmap/profiles/` as the source catalog for the initial built-in profile set.
   - Copy those YAML files into the built-in profile asset directory, preserving provider subdirectories unless the implementation has a clear reason to flatten them.
   - Do not invent a broad provider/model catalog.
   - Do not add raw API keys.
   - Built-in profiles may use `api_key_env` for CLI/HTTP compatibility when the provider requires authentication.

4. Wire repositories through a small helper.
   - Add an internal helper near adapter wiring, or in `internal/profile`, that returns:
     - built-in repository only when no custom profile source is configured;
     - overlay repository when a custom profile source is configured.
   - The runner should still receive only a `profile.Repository`.

5. Make `profile_dir` optional.
   - CLI `run`, CLI `render`, and HTTP `serve` argument/config validation should require `prompt_dir` but no longer require `profile_dir`.
   - Public `NewEngine` should no longer reject an empty `Config.ProfileDir`.
   - When `profile_dir` is empty, wire only built-ins.
   - When `profile_dir` is non-empty, wire filesystem profiles over built-ins.

### Tests

Add or update tests under `internal/adapter/cli`, `internal/adapter/http`, root public package tests, and profile/builtin tests.

Required coverage:

- CLI parse/config tests accept missing `profile_dir` when `prompt_dir` is present.
- HTTP serve parse/config tests accept missing `profile_dir` when `prompt_dir` is present.
- Public `NewEngine` accepts missing `ProfileDir`.
- A built-in profile ID can be selected with no custom `profile_dir`.
- A prompt `default_profile` can refer to a built-in profile ID.
- A custom profile in `profile_dir` overrides a built-in with the same ID.
- A malformed selected custom profile does not fall back to a built-in with the same ID.
- A missing profile ID still maps to the existing profile-not-found behavior.
- Existing duplicate-ID tests for filesystem profiles continue to fail within the custom source.

### Documentation

After code behavior exists, update non-roadmap docs:

- `docs/config.md`: document `profile_dir` as optional and describe built-in fallback/override behavior.
- `docs/cli.md`: remove claims that `--profile-dir` is required.
- `docs/internal/adapters.md`: document built-in profile repository composition.
- Any affected examples or README snippets that imply `profile_dir` is mandatory.

Do not document built-in profile IDs outside implemented assets.

### Verification

Run:

```bash
go test ./internal/profile ./internal/adapter/cli ./internal/adapter/http .
go test ./...
go run ./cmd/scriptorium render \
  --config ./examples/config.yml \
  --prompt generic.markdown_summary \
  --input transcript=./examples/fixtures/transcript.md \
  --input glossary=./examples/fixtures/glossary.yml \
  --format json
```

Also run a new smoke command that uses a built-in profile without `--profile-dir` once a concrete built-in profile ID is available.

## Stage 3: Public API Credential Value

Goal: support the production library credential model: a direct API-key Go value, without adding a resolver or accepting raw keys in serialized config.

### Public API Decision

Use a single public credential-supply method:

```go
type RunRequest struct {
	// existing fields...
	APIKey string `json:"-"`
}
```

Do not add `Config.APIKey`, `WithAPIKey`, or a credential resolver in this stage. A request-level value avoids storing secrets on long-lived engines and supports per-tenant callers.

### Implementation Steps

1. Add direct API-key plumbing through internal request/target types.
   - Add an internal direct API-key field where needed, with `json:"-"` and `yaml:"-"` tags.
   - Convert public `RunRequest.APIKey` into the internal request.
   - Carry the value to the effective execution target used for LLM generation.
   - Do not include the value in prepared/run public results, formatted output, logs, hashes, or docs examples.

2. Update credential validation.
   - If a selected/effective profile requires authentication and a direct API key is provided, do not require the environment variable to be set for the public path.
   - Preserve existing CLI/HTTP behavior that uses `api_key_env`.
   - Preserve existing errors for missing environment variables in CLI/HTTP paths.

3. Update the OpenAI-compatible LLM client.
   - Prefer the direct API-key value when present.
   - Fall back to existing `api_key_env` behavior for CLI/HTTP compatibility.
   - Never serialize or log the direct API-key value.

4. Update public LLM injection conversion.
   - Do not expose the raw API key to injected public `LLMClient` implementations unless that is strictly necessary for custom LLM execution.
   - If custom LLM clients need the key, expose it only on the public `GenerateRequest` with `json:"-"` and document that fake/test clients should avoid logging it.

### Tests

Required coverage:

- Public `Run` can call the default OpenAI-compatible client path with a direct API key without requiring the configured `api_key_env` environment variable.
- CLI/HTTP behavior using `api_key_env` still works.
- Missing credentials still fail clearly when a selected profile requires authentication and neither direct key nor usable env value is available.
- Public `PreparedRun` and `RunResult` JSON do not include the direct API key.
- Formatted prepared output does not include the direct API key.
- Direct API-key values are not included in hashes.
- Injected fake LLM tests either receive no key or receive it only through a `json:"-"` field, depending on the implementation choice above.

### Verification

Run:

```bash
go test ./internal/llm ./internal/usecase ./internal/adapter/cli ./internal/adapter/http .
go test ./...
```

## Stage 4: Public Asset Source Options

Goal: let library consumers load standard Scriptorium prompt/profile/schema assets from directories, single files, and `fs.FS` sources.

### Public API

Keep existing `Config` directory fields working. Add options:

```go
func WithPromptFS(fsys fs.FS, root string) Option
func WithPromptFile(path string) Option
func WithProfileFS(fsys fs.FS, root string) Option
func WithProfileFile(path string) Option
func WithSchemaFS(fsys fs.FS, root string) Option
func WithSchemaFile(path string) Option
```

Rules:

- Directory `Config` fields remain the compatibility path.
- Explicit options override the corresponding `Config` directory field.
- Prompt source is required.
- Profile source is optional because built-in profiles exist.
- Schema source is optional and should default to the current schema default behavior when not configured.
- Nil `fs.FS` values, empty required roots, and invalid option combinations return `ErrInvalidConfig`.

### Implementation Steps

1. Add `fs.FS` prompt-definition support.
   - Implement an `fs.FS` prompt repository that preserves existing strict prompt YAML behavior.
   - Preserve prompt lookup by YAML `id`, not path.
   - Preserve duplicate prompt ID errors within one source.
   - Resolve `content_file` relative to the prompt file's directory inside the same source.
   - Keep the existing filesystem repository API; it may delegate to the `fs.FS` implementation.

2. Add public prompt source wiring.
   - `Config.PromptDir` wires the filesystem repository.
   - `WithPromptFS` wires the `fs.FS` repository.
   - `WithPromptFile(path)` wires a single-file source and must still select by prompt YAML `id`.

3. Add public profile source wiring.
   - Reuse the Stage 1 profile `fs.FS` repository.
   - `WithProfileFS` and `WithProfileFile` become overlay primaries above built-ins.
   - Empty profile source still means built-ins only.

4. Add schema `fs.FS` support.
   - Extend or wrap the standard validator so schema files can be loaded from an `fs.FS` source.
   - `WithSchemaFS` should preserve existing `schema_path` semantics.
   - `WithSchemaFile(path)` should expose the file by its base name; prompts using it should set `schema_path` to that base name.

5. Keep adapter scope narrow.
   - This stage is for the public package and shared repositories/validators.
   - Do not change CLI/HTTP request shapes for prompt or schema `fs.FS` sources.

### Tests

Required coverage:

- Public `Prepare` works with prompt definitions from `embed.FS`.
- `content_file` references resolve relative to the prompt file in `embed.FS`.
- Public `Prepare` works with `WithPromptFile`.
- Public `Run` or `Prepare` works with `WithProfileFS` over built-ins.
- Public `Run` or `Prepare` works with `WithProfileFile` over built-ins.
- Public structured-output schema validation works with `WithSchemaFS`.
- `WithSchemaFile` works when the prompt's `schema_path` is the schema file base name.
- Explicit source options override `Config` directory fields.
- Invalid/nil source options return `ErrInvalidConfig`.
- Existing filesystem prompt/profile/schema behavior remains unchanged.

### Verification

Run:

```bash
go test ./internal/promptdef ./internal/profile ./internal/validate .
go test ./...
```

## Stage 5: Public In-Memory Profiles And Profile Templates

Goal: allow library callers to provide typed profile values and construct stable OpenAI-compatible profile templates without generating YAML.

### Public API

Add a public profile facade aligned with the profile YAML contract:

```go
type Profile struct {
	ID              string
	Endpoint        string
	Model           string
	Temperature     float64
	MaxTokens       int
	TopP            float64
	TimeoutSeconds  int
	ServiceTier     string
	ReasoningEffort string
	APIKeyRequired  bool
	ExtraParams     map[string]any
}

func WithProfiles(profiles ...Profile) Option
```

Add an OpenAI-compatible template constructor:

```go
type OpenAICompatibleProfileConfig struct {
	ID              string
	Endpoint        string
	Model           string
	APIKeyRequired  bool
	Temperature     float64
	MaxTokens       int
	TopP            float64
	TimeoutSeconds  int
	ServiceTier     string
	ReasoningEffort string
	ExtraParams     map[string]any
}

func OpenAICompatibleProfile(cfg OpenAICompatibleProfileConfig) Profile
```

Rules:

- `WithProfiles` profiles override built-ins with the same ID.
- If `WithProfiles` and a file/FS profile source are both configured, in-memory profiles have highest precedence, then file/FS profiles, then built-ins.
- Public profile values must not include raw API-key fields.
- Use `APIKeyRequired` to indicate whether `RunRequest.APIKey` is required for the public path. Internal conversion may map this to the existing auth-required/profile credential model without exposing raw keys.

### Implementation Steps

1. Add a profile repository for public in-memory `Profile` values.
   - Validate with the same effective rules as YAML profiles.
   - Reject duplicate IDs within the provided values.
   - Deep-copy `ExtraParams` across public/internal boundaries.

2. Compose profile sources in public `NewEngine`.
   - Highest: in-memory public profiles.
   - Next: configured profile file/FS/directory source.
   - Fallback: built-in profiles.

3. Add template constructor conversion.
   - `OpenAICompatibleProfile` should be a convenience constructor only.
   - It should not register global state.
   - It should not maintain a broad model catalog.

4. Ensure direct API-key behavior works with in-memory/template profiles.
   - Profiles marked `APIKeyRequired` should require `RunRequest.APIKey` for public default LLM execution.
   - Profiles not marked `APIKeyRequired` should not require an API key.

### Tests

Required coverage:

- Public `Prepare`/`Run` uses `WithProfiles` without any profile files.
- In-memory profiles override built-ins.
- In-memory profiles override file/FS profile sources when IDs collide.
- Duplicate in-memory profile IDs return `ErrInvalidConfig`.
- Template-created profiles execute through the same path as normal profiles.
- Template-created profiles requiring an API key work with `RunRequest.APIKey`.
- Template-created profiles that do not require an API key work without one.
- `ExtraParams` in public profiles are deep-copied and isolated from caller mutation.

### Verification

Run:

```bash
go test .
go test ./...
```

## Stage 6: Public Documentation And Examples

Goal: document implemented behavior in canonical locations after code exists.

### Documentation

Update:

- `README.md`: add a short pointer to library usage and built-in profiles without turning the README into a manual.
- `docs/config.md`: document optional `profile_dir`, built-in override behavior, and implemented built-in profile IDs.
- `docs/cli.md`: document optional `--profile-dir` and built-in profile selection.
- `docs/internal/adapters.md`: document repository composition and public library adapter surface.
- `docs/consumers/api.md`: describe public consumer surfaces at a high level.
- `docs/consumers/pkg-scriptorium.md`: document root package usage, source options, direct API-key value, errors, and examples.
- `docs/integrations/openai-compatible-chat.md`: update only if direct API-key plumbing changes provider-call semantics.

Do not document unimplemented roadmap behavior outside `docs/roadmap/`.

### Examples

Add or update copyable examples that require no real provider credentials:

- prepare with built-in profile and no `profile_dir`;
- public library prepare from `embed.FS`;
- public library run with injected fake LLM;
- public library run with typed/template profile and direct API key, using a fake/local provider path so no real secret is required.

Examples must be secret-free.

### Tests And Smoke Commands

Required verification:

```bash
go test ./...
go run ./cmd/scriptorium render \
  --config ./examples/config.yml \
  --prompt generic.markdown_summary \
  --input transcript=./examples/fixtures/transcript.md \
  --input glossary=./examples/fixtures/glossary.yml \
  --format json
```

Also run smoke commands for any new examples, such as:

```bash
go run ./examples/go-library/prepare
```

If an example relies on a specific built-in profile ID, use an ID from `docs/roadmap/profiles/` and include that smoke command in the implementing change.

## Final Completion Checklist

Before marking the combined feature complete:

1. `go test ./...` passes.
2. Existing CLI render smoke command passes.
3. At least one smoke command proves built-in profile lookup works without `profile_dir`.
4. Public package examples compile/run without real provider credentials.
5. `profile_dir` is optional in CLI, HTTP, and public engine construction.
6. Custom profiles override built-ins with the same ID.
7. Malformed selected custom profiles do not fall back to built-ins.
8. Built-in profiles use the same validation rules as file profiles.
9. Public `RunRequest.APIKey` is the documented library credential method.
10. Raw API keys do not appear in YAML/JSON config, prepared output, run results, logs, hashes, or examples.
11. Non-roadmap docs describe only implemented behavior.