distributor/docs/internal/storage.md

# Storage

## Purpose

`internal/storage` defines backend-rooted logical file access for core packages. Callers use slash-separated paths relative to a configured backend root.

## Inputs and outputs

The storage interface supports byte reads, stream reads, byte writes, stream writes, exact metadata lookup, traversal, destination emptiness checks, guarded managed deletion, and bounded prefix deletion for explicit forced replacement.

Entries report a logical path, type, and size when available. Entry types are `file`, `directory`, `symlink`, and `other`.

## Boundaries

Core packages should depend on `internal/storage`, not adapter packages. Adapter-specific path handling stays behind backend implementations.

The local adapter lives in `internal/adapters/local`. The SSH/SFTP adapter lives in `internal/adapters/ssh`. The S3-compatible adapter lives in `internal/adapters/s3`. Runtime backend construction is wired through the app-level backend factory and storage registry. The fake backend lives in `internal/storage/fake` for tests and is not registered for runtime use.

## Paths

Logical file paths must be non-empty, relative, clean, slash-separated, and must not contain `.` or `..` segments or backslashes. Prefix paths follow the same rules, except an empty prefix means the backend root.

## Failure behavior

Storage errors use typed categories such as not found, already exists, invalid path, conflict, permission, temporary, unsupported, and unknown. Callers should use helper predicates rather than matching error strings.

Backends may wrap implementation-specific errors, but callers should receive storage errors where practical. Traversal can stop cleanly with `ErrStopWalk`.

## Deletion

`DeleteManagedBundle` may delete listed managed outputs plus `.distributor.json`.

`DeletePrefix` removes content at and below a logical prefix for explicit forced replacement. It must not delete above the requested prefix or above the configured backend root.

## Local, SSH, S3, and fake backends

The local adapter maps logical paths to a configured filesystem root and keeps adapter-specific path handling behind the storage interface.

The SSH adapter maps logical paths to a configured remote SFTP root. It uses native SSH and SFTP libraries, supports SSH agent and key-file authentication, applies host-key policies, rejects unsafe logical paths, reports symlink entries from `Lstat`, and limits deletion to managed targets or explicit bounded prefixes.

The S3 adapter maps logical paths to object keys below a configured bucket and optional prefix. It uses the AWS SDK for Go v2, treats prefixes as object trees, requires exact objects for `Stat`, paginates traversal, applies conservative overwrite checks with `HeadObject`, infers basic content types, and limits deletion to managed target objects or explicit bounded object-key prefixes.

The fake backend is an in-memory implementation for package tests. It is not registered for runtime use.

## Tests

Before changing storage behavior, inspect tests under:

- `internal/storage`
- `internal/storage/fake`
- `internal/adapters/local`
- `internal/adapters/ssh`
- `internal/adapters/s3`

## Invariants

- Core packages depend on `internal/storage`, not concrete adapters.
- Logical paths are slash-separated and confined to the backend root.
- `storage.List` uses backend traversal and returns deterministic entries.
- Managed deletion is limited to recorded outputs plus `.distributor.json`.
- Prefix deletion is limited to the requested logical prefix.
- Runtime backend registration is owned by `internal/app`.