This document covers operating workflows, storage layout, safety behavior, and recovery. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), symptom-specific fixes live in [Troubleshooting](troubleshooting.md), and external contracts live under [Integrations](integrations/source-bundle.md).
Remote examples under `examples/ssh-destination.yml` and `examples/s3-destination.yml` are load-tested templates. Edit their endpoint, path, key, bucket, prefix, and credential values for storage you control before running them.
A source bundle is a directory containing `manifest.json` and every file listed in that manifest. See [Source Bundle Contract](integrations/source-bundle.md). Source discovery walks beneath the configured source backend root and finds bundle directories.
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple bundles have the same timestamp, the source-root-relative bundle path in ascending order wins.
Published destination bundle paths contain `.distributor.json`. See [Destination State Contract](integrations/destination-state.md). This file is both the managed sentinel and the destination state record. It records the pipeline id, destination id, publication time, source manifest, copied outputs, generated outputs, and optional public URL metadata.
Normal replacement deletes only managed output paths recorded in `.distributor.json` plus the state file, then verifies the destination bundle path is empty before writing new outputs and state.
If a write fails after some outputs were written, `distributor` attempts to delete outputs from that failed attempt so a retry does not treat partial outputs as unmanaged content. Operators should still inspect the destination after a failed write before retrying.
Fan-out destinations are independent. If one destination fails after planning or execution begins, later destinations are still attempted. The command exits non-zero if any destination failed.
`run --dry-run` loads config, resolves credentials, discovers source bundles, opens destinations, inspects destination state, builds publish plans, and prints actions. It does not write outputs, `.distributor.json`, or SSH `known_hosts` entries.
Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.
JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
Forced replacement can claim unmanaged non-empty destination paths. State conflicts require both `--force` and transfer policy that permits replacement:
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.
The [HTTP Upload API Contract](integrations/http-upload.md) defines request and response details. `distributor serve` runs the HTTP upload API for pipelines whose source backend is `http_upload`. Each bearer token maps to exactly one configured upload pipeline. Token values come from the process environment or `secrets.directory`, not from YAML literal values.
Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire after `server.http.retention`. Expiration removes committed staged bundle directories for completed uploads. Restarting the process clears upload status, queue state, and in-memory records.
Upload admission is bounded by `server.http.queue_size`. Publication concurrency is bounded by `server.http.max_concurrency`, and the coordinator does not run two uploads for the same pipeline at the same time.
The upload server accepts `application/x-tar`, `application/gzip`, and `application/x-gzip`. Archives are extracted into a temporary staging directory, must contain exactly one root-level `manifest.json`, and must validate as one complete source bundle before a run id is issued. Per-source `max_upload_size` bounds both uploaded archive size and extracted bundle size. The implementation also caps extracted file count.
The default bind address is private loopback. Put TLS, public routing, rate limiting, and external access policy in a reverse proxy or deployment layer.
Configure `ssh_key_file`, an SSH agent, or both. Agent identities are attempted first, followed by the configured key file. YAML password authentication is not supported.
The default host key policy is `accept-new`. During dry runs, unknown host keys may be accepted for the current connection but are not persisted. Changed host keys are fatal for `strict` and `accept-new`. `host_key_policy: off` disables host key checking and should be limited to controlled test environments.
S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration](integrations/s3.md). Configure an endpoint, bucket, optional prefix, optional region, optional path-style setting, and optional explicit credential variable names.
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
`secrets.directory` is loaded during `run`, `serve`, and configured-source `validate` or `inspect` before credential-consuming work starts. If the directory is missing, unreadable, or contains an invalid secret filename, the command fails before storage work starts.
Real process environment values take precedence over files with the same name. If the values differ and stdout is enabled, commands emit a warning naming the ignored secret variable without printing either value. The process environment is not modified.
- For source validation failures, regenerate the source bundle and manifest together.
- For an empty or missing destination, rerun after fixing config or storage access.
- For unmanaged destination content, move unrelated files aside or use a different destination path before publishing.
- For failed writes, inspect the destination bundle path, remove only confirmed partial outputs if needed, then rerun `--dry-run`.
- For state conflicts, verify the source, pipeline, destination, and existing `.distributor.json` before considering `--force`.
- For HTTP upload failures, inspect `/runs/<run-id>` while retained; after expiry or restart, rely on destination state and logs/output from the publishing run.
