distributor/docs/troubleshooting.md

# Distributor Troubleshooting

Audience: administrators and operators diagnosing `distributor` command, configuration, publishing, storage, or HTTP upload failures.

Each entry lists the symptom, likely cause, diagnostic step, safe fix, and relevant reference link. Command syntax lives in [CLI](cli.md), configuration fields live in [Configuration](config.md), and operating procedures live in [Operations](operations.md).

## Config File Is Missing

Symptom: `load config ... no such file or directory`.

Likely cause: `--config` points to a missing file, or `--config` was omitted and `/usr/local/etc/distributor/config.yml` is not installed.

Diagnostic:

```sh
ls -l <config-path>
```

Safe fix: pass an existing file with `--config`, or install a config at the default path.

Reference: [Configuration](config.md#config-file-loading).

## Config Contains An Unknown Field

Symptom: `parse config ... field not found`.

Likely cause: the YAML contains a key that is not part of the implemented config schema. Legacy destination policy keys `state`, `reconciliation`, `takeover`, and `transfer` are rejected.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: remove unsupported fields using the canonical config reference. Destination behavior is configured with `workflow`, `publish`, `transform`, `path_mapping`, `links`, and `retention`.

Reference: [Configuration](config.md#destination-fields).

## Backend Name Or Placement Is Invalid

Symptom: `backend ... is unsupported` or `http_upload is only supported for sources`.

Likely cause: a backend name is misspelled, not executable, or configured in the wrong role.

Diagnostic:

```sh
rg -n "backend:" <config-path>
```

Safe fix: use `local`, `ssh`, or `s3` for executable sources and destinations. Use `http_upload` only as a source served by `distributor serve`.

Reference: [Configuration](config.md#backend-reference).

## CLI Arguments Select The Wrong Source Mode

Symptom: `configured source mode requires --pipeline`, `does not accept a local path with --config, --pipeline, or --bundle`, `validate command requires a path`, or `inspect command requires a path`.

Likely cause: `validate` or `inspect` mixed local path mode with configured source mode, or omitted the required source selector.

Diagnostic:

```sh
go run ./cmd/distributor validate --help
go run ./cmd/distributor inspect --help
```

Safe fix: use either `distributor validate <path>` / `distributor inspect <path>`, or use `--config <path> --pipeline <id>` with optional `--bundle <path>`.

Reference: [CLI](cli.md#validate).

## Reconcile-State Selector Is Missing Or Wrong

Symptom: `reconcile-state requires --config`, `requires --pipeline`, `requires --destination`, `pipeline "<id>" not found`, or `destination <id> not found`.

Likely cause: the command did not identify one configured destination root.

Diagnostic:

```sh
go run ./cmd/distributor reconcile-state --help
rg -n 'pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```

Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file.

Reference: [CLI](cli.md#reconcile-state).

## Prune Selector Or Mode Is Missing Or Wrong

Symptom: `prune requires --config`, `requires --pipeline`, `requires --destination`, `requires exactly one of --dry-run or --apply`, `pipeline "<id>" not found`, or `destination <id> not found`.

Likely cause: the command did not identify one configured destination root or did not choose exactly one execution mode.

Diagnostic:

```sh
go run ./cmd/distributor prune --help
rg -n 'retention:|prune:|pipelines:|destinations:|id:' <config-path>
cat <destination-path>/.distributor.json
```

Safe fix: pass the configured `--config`, `--pipeline`, and `--destination` values that identify the destination root containing the state file. Use `--dry-run` for read-only review or `--apply` for deletion, but not both.

Reference: [CLI](cli.md#prune).

## Output Format Is Invalid

Symptom: `format must be text or json`.

Likely cause: an unsupported value was passed to `--format`.

Diagnostic:

```sh
go run ./cmd/distributor run --help
```

Safe fix: use `--format text` or `--format json`.

Reference: [CLI](cli.md#common-output-format).

## JSON Mode Wrote No JSON Document

Symptom: `--format json` exits non-zero and stdout has no JSON result.

Likely cause: the command failed before it could construct a result, such as invalid arguments, missing config, unreadable secrets, or source setup failure.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: read stderr, fix the setup problem, then rerun. Partial destination failures during `run` can produce JSON; fatal setup failures do not.

Reference: [CLI](cli.md#output-and-exit-behavior).

## Source Pipeline Is Not Found

Symptom: `pipeline "<id>" not found`.

Likely cause: configured source diagnostics or upload processing selected a pipeline id that is absent from the loaded config.

Diagnostic:

```sh
rg -n "id:" <config-path>
```

Safe fix: pass an existing `--pipeline` value or correct the pipeline id in config.

Reference: [Configuration](config.md#pipelines).

## Source Bundles Are Not Found

Symptom: `no bundles found`, `no bundles found under "."`, `stat ssh ... not_found`, or `stat s3 ... not_found`.

Likely cause: the source root, source-root-relative bundle path, S3 prefix, SSH path, or permissions do not expose a directory containing `manifest.json`.

Diagnostic:

```sh
go run ./cmd/distributor validate --config <config-path> --pipeline <pipeline-id>
```

Safe fix: correct the configured source backend root, permissions, prefix, or `--bundle` path. The selected bundle directory must contain `manifest.json`.

Reference: [Operations](operations.md#filesystem-and-storage-layout).

## Source Manifest Or Files Fail Validation

Symptom: `sha256 mismatch`, `size mismatch`, `digest mismatch`, missing manifest fields, or unsafe source paths.

Likely cause: files changed after `manifest.json` was written, the manifest digest is stale, or the producer wrote invalid bundle paths.

Diagnostic:

```sh
go run ./cmd/distributor validate <source-root>
```

Safe fix: regenerate the producer bundle and manifest together. Do not edit destination state to work around source validation failures.

Reference: [Operations](operations.md#cleanup-and-recovery).

## Destination Has Unmanaged Content

Symptom: `destination has content but no distributor state`, `destination output path ... exists but is not managed by catalog state`, or a plan reason containing `fail_unmanaged`.

Likely cause: the destination bundle path contains files but no valid `.distributor.json`, or a planned output path collides with storage content that valid catalog state does not record. `distributor` will not claim unmanaged content by default.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: choose an empty destination path, move unrelated files aside, or preview `run --dry-run --force` only after confirming the reported destination bundle path is safe to replace.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Destination State Is Invalid Or Unsupported

Symptom: `fail_conflict`, `parse distributor state`, `state schema_version must be 4`, or `unsupported future destination state`.

Likely cause: `.distributor.json` is invalid JSON, has invalid catalog fields, or uses an unsupported future schema.

Diagnostic:

```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run --format json
```

Safe fix: restore a valid catalog state file from backup, choose a different destination path, or use `--force` only after `run --dry-run --force` reports the intended bounded `force_replace`.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Destination Uses Superseded Legacy State

Symptom: dry-run reports a normal catalog action against an older `.distributor.json`, or `reconcile-state` / `prune` reports that the destination state schema is superseded.

Likely cause: the destination contains a state file written by an older implementation. Publish planning can replace it with catalog state on successful `run`, but maintenance commands only operate on current catalog state.

Diagnostic:

```sh
cat <destination-path>/.distributor.json
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: preview the publish plan, then run publication if the destination path is correct. The successful run writes schema version `4` catalog state.

Reference: [Destination State Contract](integrations/destination-state.md).

## Destination State References Missing Managed Outputs

Symptom: `reconcile-state --dry-run` reports `status=would_change` or JSON `missing_managed_outputs` entries.

Likely cause: files that were recorded as managed in `.distributor.json` were removed outside `distributor`, or a previous external cleanup removed destination files without updating state.

Diagnostic:

```sh
go run ./cmd/distributor reconcile-state \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run
```

Safe fix: if the missing files should no longer be managed, rerun the same command without `--dry-run` to remove only the missing managed output records from `.distributor.json`. The command does not delete destination files or adopt unmanaged entries. If the files should exist, restore them from backup or republish instead.

Reference: [Operations](operations.md#destination-state-repair).

## Prune Reports No Planned Deletes

Symptom: `prune --dry-run` reports `planned=0` or JSON `planned_outputs: []`.

Likely cause: pruning is disabled for the selected destination, every managed output is preserved by `keep_latest`, no managed output is older than `older_than`, or the selected pipeline/destination owner has no eligible outputs.

Diagnostic:

```sh
rg -n 'retention:|prune:|older_than:|keep_latest:' <config-path>
go run ./cmd/distributor prune \
  --config <config-path> \
  --pipeline <pipeline-id> \
  --destination <destination-id> \
  --dry-run \
  --format json
```

Safe fix: verify the selected destination's retention policy and the managed output `updated_at` timestamps in `.distributor.json`. Adjust config if the policy is too conservative, then rerun `--dry-run` before using `--apply`.

Reference: [Operations](operations.md#managed-output-pruning).

## Forced Replacement Appears In A Plan

Symptom: dry-run output includes `force_replace`.

Likely cause: the run used `--force`, and catalog planning selected a supported destructive replacement for unmanaged destination content, a planned unmanaged path collision, invalid destination state, or unsupported future destination state.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run --force
```

Safe fix: inspect the pipeline id, destination id, backend, and bundle path. Confirm the `destination_path` in JSON output, or the fixed-path `target=.` text output, before applying. Proceed only if deleting everything inside that destination bundle path is intended; `force_replace` then writes planned outputs and schema version `4` catalog state.

Reference: [Operations](operations.md#forced-replacement-workflow).

## Output Path Collision

Symptom: `destination output path collision` or `destination output path ... exists but is not managed by catalog state`.

Likely cause: publication would write two outputs to the same destination path, such as copying `report.html` while also generating `report.html` from `report.md`, or a planned output path already exists in storage but is not catalog-managed.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --dry-run
```

Safe fix: adjust source files or publish/transform policy so copied and generated outputs do not collide. For unmanaged storage collisions, move the unmanaged file aside, choose another destination path, or use forced replacement only when deleting the destination bundle path is intended.

Reference: [Configuration](config.md#publish-and-transform-policy).

## Run Failed After Writing Some Files

Symptom: a destination write failed and the command exited non-zero after partial work.

Likely cause: storage write failure, permission issue, network interruption, or object-store error during publish execution.

Diagnostic:

```sh
find <destination-path> -maxdepth 2 -print
```

Safe fix: inspect the destination bundle path printed in the error. `distributor` attempts to remove outputs from the failed attempt, but operators should verify the destination before retrying. Rerun `--dry-run` before publishing again.

Reference: [Operations](operations.md#catalog-publish-behavior).

## JSON Run Result Has `ok: false`

Symptom: `run --format json` exits non-zero with a JSON result where `ok` is `false`.

Likely cause: at least one destination failed after planning or execution began, while other destination results were still available.

Diagnostic:

```sh
go run ./cmd/distributor run --config <config-path> --format json
```

Safe fix: inspect the top-level `errors` array, destination actions, output errors, and summary. Fix failed destinations, then preview with `--dry-run --format json`.

Reference: [CLI](cli.md#output-and-exit-behavior).

## Secrets Directory Is Missing Or Unreadable

Symptom: `load secrets directory ... no such file or directory`, `permission denied`, or `secret filename ... is invalid`.

Likely cause: `secrets.directory` points to a missing or unreadable directory, or contains a filename that cannot be used as a credential variable name.

Diagnostic:

```sh
ls -la <secrets-directory>
```

Safe fix: create the directory, fix permissions, or rename secret files to valid environment-variable-style names.

Reference: [Configuration](config.md#secrets).