distributor/docs/policy/development.md

# Development Policy

This document defines the day-to-day development workflow for `distributor`.
Use it with `docs/policy/architecture.md` and `docs/policy/documentation.md`.

## Repository Layout

- `cmd/distributor`: executable entrypoint only.
- `internal/app`: top-level use cases for `run`, `validate`, and `inspect`.
- `internal/cli`: standard-library command parsing, flags, help text, and command wiring.
- `internal/config`: YAML configuration structs, loading, defaults, and validation.
- `internal/bundle`: source bundle discovery, manifest parsing, digest calculation, and validation.
- `internal/state`: destination `.distributor.json` parsing, validation, and comparison.
- `internal/storage`: backend interface, registry, logical path rules, typed errors, and shared storage helpers.
- `internal/adapters/local`: local filesystem backend.
- `internal/adapters/ssh`: SSH/SFTP backend.
- `internal/adapters/s3`: S3-compatible object storage backend.
- `internal/storage/fake`: in-memory backend for tests.
- `internal/publish`: destination inspection, output planning, reconciliation, execution, managed cleanup, and explicit forced replacement.
- `internal/transform`: transform interface and registry.
- `internal/transform/markdown`: Markdown-to-HTML sidecar transform.
- `internal/notify`: notification interface and current no-op notifier.
- `internal/testutil`: shared test fixtures. Production code must not import this package.
- `docs`: current user, operator, policy, internal, and roadmap documentation.
- `examples`: copyable example configs and source bundles.

Do not create new top-level package families such as `pkg`, `internal/stage`,
`internal/modules`, or service-specific adapter directories unless the
architecture policy or a current roadmap explicitly calls for them.

## Common Commands

Run the full test suite:

```bash
go test ./...
```

Run targeted packages while developing:

```bash
go test ./internal/config
go test ./internal/cli ./internal/app
go test ./internal/publish ./internal/state
go test ./internal/transform/markdown
```

Run the CLI against an example config:

```bash
go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
```

Validate or inspect a local source bundle:

```bash
go run ./cmd/distributor validate examples/source-bundle
go run ./cmd/distributor inspect examples/source-bundle
```

If Go cache permissions fail in a restricted environment, use workspace-safe
temporary caches:

```bash
GOCACHE=/private/tmp/distributor-gocache GOMODCACHE=/private/tmp/distributor-gomodcache go test ./...
```

## Coding Conventions

- Keep the application small, explicit, and dependency-light.
- Prefer package-local helpers over broad abstractions until behavior is shared by multiple packages.
- Keep CLI parsing in `internal/cli`; business decisions belong in `internal/app`, `internal/bundle`, `internal/publish`, `internal/state`, and related core packages.
- Keep adapter packages thin. Backend-specific filesystem or service behavior belongs in adapters; bundle, state, transform, and publish policy belongs outside adapters.
- Preserve public CLI behavior, config semantics, manifest schema, destination state schema, and implemented backend behavior unless the current task explicitly changes them.
- Use `storage.DisplayPath`, `storage.StateFileName`, `storage.StatePath`, and `storage.ManagedBundleTargets` instead of duplicating those conventions.
- Use `bundle.ValidateManifest` for normalized source manifest semantics, including embedded source manifests in destination state.
- Use `config.ValidatePublishTransformPolicy` for publish and transform policy combinations.
- Do not import concrete transform implementations from `internal/publish`; app-level wiring owns transform registration.
- Do not import `internal/testutil` from production code.

## Dependency Policy

The project currently depends on:

- `gopkg.in/yaml.v3` for YAML configuration loading.
- `github.com/yuin/goldmark` for Markdown rendering.
- `golang.org/x/crypto/ssh`, `golang.org/x/crypto/ssh/agent`, and `golang.org/x/crypto/ssh/knownhosts` for native SSH support.
- `github.com/pkg/sftp` for native SFTP support.
- `github.com/aws/aws-sdk-go-v2/...` packages for S3-compatible storage support.

Add external dependencies only when they materially improve correctness,
security, interoperability, or implementation complexity. Avoid dependencies
for small conveniences. Do not let dependency-specific types leak across
internal package boundaries unless that dependency is the explicit package
contract.

## Configuration Changes

When adding or changing configuration:

1. Update `internal/config/config.go` structs and YAML tags.
2. Add defaults in `internal/config/defaults.go` only for built-in defaults.
3. Add validation in `internal/config/validate.go` with clear field context.
4. Update config load and validation tests.
5. Update `docs/config.md` in the same change if current user-visible config behavior changes.
6. Update examples only with configs that are valid and executable for implemented behavior.

Config validation may accept fields for roadmap backends before execution
support exists, but user-facing docs and examples must clearly state execution
support. Runtime executable backends are local, SSH, and S3.

Credential-consuming code must use the config-owned environment resolver for
explicit credential environment variable references. Do not call `os.Getenv`
directly for backend credentials, because `secrets.directory` values are
intentionally available through the resolver without mutating the process
environment.

## CLI Changes

The CLI is hand-written with the Go standard library. Do not introduce a CLI
framework without a documented reason.

When adding or changing commands or flags:

1. Keep parsing and help text in `internal/cli`.
2. Keep command work in `internal/app` or a lower-level package.
3. Add or update CLI tests in `internal/cli`.
4. Update `docs/cli.md` if syntax, flags, output expectations, or workflows change.

`validate` and `inspect` are local path commands. `run` loads configured
pipelines and executes local, SSH, and S3 backends.

## Storage Backends

Storage behavior is defined by `internal/storage.Backend` and shared path rules
in `internal/storage`.

When adding a backend:

1. Implement the storage interface in an adapter package.
2. Translate backend-specific errors into storage errors where practical.
3. Keep bundle comparison, transform, routing, and replacement policy out of the adapter.
4. Register runtime construction through app-level backend factory wiring.
5. Add focused adapter tests and app-level wiring tests.
6. Update user docs, operations docs, examples, and internal docs only for behavior that is actually implemented.

Do not document future backend execution as available until the corresponding
adapter package and app wiring exist.

## Transforms

Transforms use `internal/transform` interfaces and registry wiring.

When adding or changing a transform:

1. Keep the transform implementation in its own package under `internal/transform`.
2. Register default runtime transforms from `internal/app`.
3. Keep `internal/publish` dependent only on the transform interface or resolver.
4. Record deterministic output metadata: path, source path, transform name, digest, and size.
5. Add transform tests and app or publish tests for wiring and policy behavior.
6. Update `docs/internal/transform.md` and any relevant integration docs for implemented behavior.

## Tests

Test close to the behavior being changed:

- Use package tests for parsing, validation, comparison, planning, and adapter behavior.
- Use `internal/app` and `internal/cli` tests for user-facing workflows.
- Use `internal/testutil` for shared valid fixtures only; keep edge cases near the package under test.
- Run `go test ./...` after cross-package changes or documentation/example changes tied to tests.

Live integration tests must be opt-in and skipped during normal `go test ./...`
unless their required environment variables are set. Test-only environment
variables must use this prefix shape:

```text
DISTRIBUTOR_TEST_<BACKEND>_*
```

Examples include `DISTRIBUTOR_TEST_SSH_HOST` and
`DISTRIBUTOR_TEST_S3_ENDPOINT`. Do not use production credential variable names
for test-only controls.

## Examples

Examples under `examples/` must be valid, maintained, and free of secrets.
They should be copyable for implemented behavior. Remote examples must use
placeholders or environment variables for endpoint and credential material.

When changing examples:

1. Keep paths relative to the repository where practical.
2. Keep local examples runnable without external services; gate remote examples
   behind user-provided endpoints and credentials.
3. Run `go test ./internal/config` because config tests load examples.
4. Update README, CLI, or config docs if links or recommended workflows change.

## Documentation

Follow `docs/policy/documentation.md`.

- Document implemented behavior outside `docs/roadmap/`.
- Keep future, planned, or aspirational behavior under `docs/roadmap/`.
- Keep `docs/config.md` canonical for user-facing config reference.
- Keep `docs/cli.md` canonical for command syntax and workflows.
- Keep `docs/operations.md` canonical for operational and recovery behavior.
- Keep `docs/internal/` focused on implemented package contracts.
- Update docs in the same change as behavior when public behavior, config, CLI, examples, or internal contracts change.