distributor/docs/troubleshooting.md

# Distributor Troubleshooting

## `load config ... no such file or directory`

Likely cause: `run` could not find the config path. If `--config` is omitted, the default path is `/usr/local/etc/distributor/config.yml`.

Diagnostic:

```sh
ls -l <config-path>
```

Safe fix: pass an existing config path with `--config`, or install a config at the default path. See [configuration](config.md).

## `parse config ... field not found`

Likely cause: the YAML contains an unknown field. Config loading rejects unknown keys.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: compare the file to the reference in [configuration](config.md) and remove or rename unsupported fields.

## `validate config ... backend ... is unsupported`

Likely cause: a source or destination uses an unsupported backend name, or a
command is trying to execute a backend that is valid only for another workflow.
`run`, `validate`, and `inspect` execute `local`, `ssh`, and `s3` sources.
`serve` executes `http_upload` sources.

Diagnostic:

```sh
rg -n "backend:" <config-path>
```

Safe fix: use `backend: local`, `backend: ssh`, or `backend: s3` for normal
source and destination workflows. Use `backend: http_upload` only for sources
handled by `distributor serve`.

## `bind HTTP server ... address already in use`

Likely cause: another process is already listening on `server.http.bind`.

Diagnostic:

```sh
ss -ltnp | rg '<port>'
```

Safe fix: stop the conflicting process or configure a different
`server.http.bind` value. The default bind address is `127.0.0.1:8080`.

## `upload token environment variable ... is not set`

Likely cause: a configured `http_upload` source references `token_env`, but the
variable is absent from both the real process environment and
`secrets.directory`.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^<token-variable>$'
ls -l <secrets-directory>/<token-variable>
```

Safe fix: set the real environment variable or create a readable
secrets-directory file with the same name. Do not place literal token values in
YAML.

## `upload token environment variables ... resolve to the same value`

Likely cause: two configured `http_upload` pipelines resolve to the same bearer
token value.

Diagnostic:

```sh
rg -n 'token_env:' <config-path>
```

Safe fix: assign a distinct non-empty token value to each `http_upload`
pipeline. Distributor does not print the duplicate token value.

## `POST /upload` returns `401`

Likely cause: the request is missing `Authorization: Bearer <token>` or the
token does not match any configured `http_upload` pipeline.

Diagnostic:

```sh
curl -i -X POST http://127.0.0.1:8080/upload \
  -H "Authorization: Bearer $DISTRIBUTOR_UPLOAD_TOKEN" \
  -H "Content-Type: application/x-tar" \
  --data-binary @bundle.tar
```

Safe fix: use the token value resolved by the configured `token_env`. Do not
include token values in logs or tickets.

## `POST /upload` returns `413`

Likely cause: the request body exceeds the selected pipeline's
`source.max_upload_size` or the default `server.http.max_upload_size`.

Diagnostic:

```sh
ls -lh bundle.tar bundle.tar.gz
rg -n 'max_upload_size:' <config-path>
```

Safe fix: upload a smaller archive, remove unnecessary files from the source
bundle, or raise the configured upload size limit.

## `POST /upload` returns `415`

Likely cause: the upload uses an unsupported content type. The server accepts
uncompressed tar and gzip-compressed tar archives only.

Diagnostic:

```sh
file bundle.tar.gz
```

Safe fix: send `Content-Type: application/x-tar`, `application/gzip`, or
`application/x-gzip`, matching the archive format.

## `POST /upload` returns `503`

Likely cause: the in-memory upload queue is full.

Diagnostic:

```sh
rg -n 'queue_size|max_concurrency' <config-path>
```

Safe fix: retry after active uploads finish, or increase `server.http.queue_size`
for the deployment.

## `GET /runs/<run_id>` returns `404`

Likely cause: the run id is wrong, the process restarted, or the completed
status record expired after `server.http.retention`.

Diagnostic:

```sh
curl -i http://127.0.0.1:8080/runs/<run-id>
rg -n 'retention:' <config-path>
```

Safe fix: use the exact `run_id` returned by `POST /upload`. If status retention
is too short for operators, increase `server.http.retention`.

## `--format: format must be text or json`

Likely cause: a command was run with an unsupported output format.

Diagnostic:

```sh
go run ./cmd/distributor run --help
```

Safe fix: use `--format text` or `--format json`. Help and usage output are always text.

## `--format json` wrote no JSON output

Likely cause: the command failed before it could construct a result, such as a missing config file, invalid arguments, unreadable secrets directory, or source setup failure.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: read the stderr error and fix the setup problem. JSON mode writes a document only after the command has enough information to construct a result.

## `configured source mode requires --pipeline`

Likely cause: `validate` or `inspect` was run with `--config` but without an explicit pipeline id.

Diagnostic:

```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```

Safe fix: add `--pipeline <pipeline-id>`. Configured source diagnostics require an explicit pipeline even when the config contains one pipeline.

## `does not accept a local path with --config, --pipeline, or --bundle`

Likely cause: local-path mode and configured source mode were mixed in one `validate` or `inspect` command.

Diagnostic:

```sh
go run ./cmd/distributor inspect --help
```

Safe fix: use either `distributor inspect <local-path>` or `distributor inspect --config <path> --pipeline <id>`, not both.

## `--format json` exited non-zero with `ok: false`

Likely cause: `run` began planning or executing destinations, and at least one destination failed while other destination results were still available.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: inspect the top-level `errors` array, `result.actions`, and `result.summary`. Fix the failed destination, then preview with `--dry-run --format json` before retrying.

## `prefix must be a clean relative slash-separated path`

Likely cause: S3 `prefix` contains traversal, dot segments, empty segments, or backslashes after leading and trailing slashes are trimmed.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: use a clean relative prefix such as `reports/archive`, or omit `prefix`.

## `NoSuchBucket`, `InvalidBucketName`, or `not_found`

Likely cause: the S3 bucket, endpoint, or prefix is wrong, or the configured credentials cannot see the requested object.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: verify `endpoint`, `bucket`, `region`, `force_path_style`, and `prefix`. For S3-compatible services, keep `force_path_style: true` unless the service requires virtual-host addressing.

## `AccessDenied`, `InvalidAccessKeyId`, or `SignatureDoesNotMatch`

Likely cause: S3 credentials are missing, wrong, empty, or lack permission for the bucket or prefix.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^(<access-key-variable>|<secret-key-variable>)$'
ls -l <secrets-directory>
```

Safe fix: provide both configured credential environment variables through the real environment or `secrets.directory`, or omit explicit credential fields to use the AWS SDK default credential chain.

## S3 endpoint connection failures

Likely cause: the endpoint URL is unreachable, uses the wrong scheme, or does not match the configured path-style mode.

Diagnostic:

```sh
curl -I <endpoint>
```

Safe fix: correct `endpoint`, network routing, TLS settings outside distributor, or `force_path_style`. Distributor does not provide insecure TLS bypass configuration.

## `load secrets directory ... no such file or directory`

Likely cause: `secrets.directory` points to a missing directory.

Diagnostic:

```sh
ls -ld <secrets-directory>
```

Safe fix: create or mount the directory before running, or remove `secrets.directory` if no credential files are needed.

## `load secrets directory ... permission denied`

Likely cause: the service user cannot read the configured secrets directory.

Diagnostic:

```sh
ls -ld <secrets-directory>
namei -l <secrets-directory>
```

Safe fix: adjust the directory path or deployment permissions so the service user can read the directory. Distributor does not enforce owner, group, or mode policy beyond OS read access.

## `secret filename ... is invalid`

Likely cause: a regular file in `secrets.directory` does not match `[A-Za-z_][A-Za-z0-9_]*`.

Diagnostic:

```sh
find <secrets-directory> -maxdepth 1 -type f -printf '%f\n'
```

Safe fix: rename the file to a valid credential environment variable name, or remove it from the secrets directory.

## `credential environment variable ... is not set`

Likely cause: a backend credential field references an environment variable that is absent from both the real process environment and the configured secrets directory.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```

Safe fix: set the real environment variable or create a readable secrets-directory file with the same name.

## `secret ... ignored because the real environment already has that variable`

Likely cause: the real process environment and secrets directory both define the variable with different values.

Diagnostic:

```sh
env | cut -d= -f1 | rg '^<variable-name>$'
ls -l <secrets-directory>/<variable-name>
```

Safe fix: remove one source of the credential or make the deployment intentionally prefer the real environment value. Distributor does not print either value.

## `host is required for ssh backend`

Likely cause: SSH config is missing the structured `host` field, or an old URL-style SSH config is still in use.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: configure SSH with `host`, optional `user` and `port`, and `path`. SSH URLs are not part of the active config schema.

## `no SSH auth methods configured`

Likely cause: neither an SSH agent nor `ssh_key_file` is available.

Diagnostic:

```sh
test -n "$SSH_AUTH_SOCK" && ssh-add -l
ls -l <ssh-key-file>
```

Safe fix: start an SSH agent with an appropriate key loaded, or configure `ssh_key_file` with a readable private key.

## `host key ... is unknown` or `known_hosts is required`

Likely cause: strict host key checking has no known host key, or `accept-new` cannot persist a new key.

Diagnostic:

```sh
ls -l <known-hosts-path>
ssh-keygen -F <host> -f <known-hosts-path>
```

Safe fix: configure a writable `known_hosts` path for `accept-new`, pre-populate `known_hosts` for `strict`, or explicitly use `host_key_policy: off` only for insecure test environments.

## `host key ... has changed`

Likely cause: the remote server presented a different host key than the one recorded in `known_hosts`.

Diagnostic:

```sh
ssh-keygen -F <host> -f <known-hosts-path>
```

Safe fix: verify the server identity out of band before updating `known_hosts`. Do not switch to `host_key_policy: off` to bypass an unexpected changed key.

## `pipeline "<id>" not found`

Likely cause: configured source validation or inspection requested a pipeline id that is not present in the config file.

Diagnostic:

```sh
rg -n "id:" <config-path>
```

Safe fix: pass an existing pipeline id with `--pipeline`, or update the config.

## `stat ssh ... not_found`, `stat s3 ... not_found`, or `no bundles found`

Likely cause: the configured source root is wrong, unreadable, or does not contain source bundles.

Diagnostic:

```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
```

Safe fix: correct the configured source root, S3 prefix, permissions, or source bundle location. Use `--bundle <path>` only with a source-root-relative bundle directory that contains `manifest.json`.

## `validate command requires a path` or `inspect command requires a path`

Likely cause: `validate` or `inspect` was run without a local path and without configured source mode.

Diagnostic:

```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```

Safe fix: pass a local source bundle directory or local tree, or pass both `--config <path>` and `--pipeline <id>`.

## `no bundles found under "."`

Likely cause: the selected source root does not contain a `manifest.json` source bundle.

Diagnostic:

```sh
find <source-root> -name manifest.json -print
```

Safe fix: point the command or config at the directory containing the source bundle, or write a valid `manifest.json` and listed files. See [CLI](cli.md).

## `sha256 mismatch`, `size mismatch`, or `digest mismatch`

Likely cause: a listed source file changed after `manifest.json` was created, or the manifest digest does not match its file list.

Diagnostic:

```sh
go run ./cmd/distributor validate <source-root>
```

Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source digest failures.

## `destination has content but no distributor state`

Likely cause: the destination path is not empty and has no `.distributor.json` state file, so `distributor` will not claim it as managed.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: choose an empty destination path or move existing files aside after confirming they are not needed. If the destination should be claimed by distributor, preview with `run --dry-run --force` and publish with `run --force` only after confirming the reported `force_replace` action is bounded to the intended bundle path.

## `fail_conflict`

Likely cause: existing `.distributor.json` belongs to a different pipeline, a different destination, a different source id, or a same-created source with a different digest.

Diagnostic:

```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor inspect <source-root>
```

Safe fix: verify you are publishing the intended source to the intended destination. Use a separate destination path for unrelated content. If the existing state should be replaced, configure `transfer.on_conflict: replace`, preview with `run --dry-run --force`, then publish with `run --force`.

## `destination is newer and replacement requires --force`

Likely cause: config explicitly allows newer-destination replacement, but the current run did not include `--force`.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
```

Safe fix: prefer the default `transfer.on_destination_newer: skip` unless replacing newer destination state is intentional. To replace it, keep `transfer.on_destination_newer: replace`, confirm the dry-run output shows `force_replace`, then run with `--force`.

## `force_replace`

Likely cause: the current run used `--force` and publish planning selected a supported destructive replacement.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
```

Safe fix: inspect the printed pipeline id, destination id, backend, and bundle path. Proceed only if deleting all content within that destination bundle path is intended.

## `destination output path collision`

Likely cause: configured publication would write two outputs to the same destination path, such as publishing a source `report.html` while also generating `report.html` from `report.md`.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: adjust the source bundle contents or publish policy so source and generated outputs do not collide.

## A run failed after writing some files

Likely cause: a write failed partway through publication. Local, SSH, and S3 execution attempt to clean up outputs written during the failed attempt.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: use the pipeline id, destination id, backend, and bundle path printed in the run error to inspect the destination before retrying. If only unrelated unmanaged files remain, move them aside or choose a clean destination. Re-run with `--dry-run` before publishing again. See [operations](operations.md).