distributor/docs/consumers/pkg-upload.md

# `pkg/upload`

Audience: upstream Go producer developers and LLM coding agents submitting bundles to `distributor serve`.

Import path:

```go
import "gitea.maximumdirect.net/eric/distributor/pkg/upload"
```

`pkg/upload` is the producer-facing HTTP upload client. It builds on `pkg/bundle`, packages valid source bundles as gzip-compressed tar archives, sends bearer authentication, routes uploads to a configured pipeline, includes idempotency keys, and exposes a status polling helper.

`UploadFiles` examples also use:

```go
import "gitea.maximumdirect.net/eric/distributor/pkg/bundle"
```

The canonical HTTP wire contract is [HTTP Upload API Contract](../integrations/http-upload.md).

## Client Construction

```go
client, err := upload.NewClient(upload.ClientOptions{
	Endpoint: "https://distributor.example.com",
	Token:    token,
})
if err != nil {
	return err
}
```

`Endpoint` is the distributor server base URL. The client derives `/v1/pipelines/<pipeline-id>/upload` and `/runs/<run-id>`. `Token` is required and is sent as `Authorization: Bearer <token>`. Token values are redacted from client errors.

`HTTPClient` and `Retry` are optional. Defaults use a 30 second HTTP timeout and safe retry settings.

## Upload Producer Files

Use `UploadFiles` when the producer has generated output files but has not assembled a bundle directory.

```go
result, err := client.UploadFiles(ctx, upload.UploadFilesOptions{
	PipelineID:     "weather-hourly",
	ID:             "weather.hourly.brentwood",
	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
	Files: []bundle.BundleFile{
		{SourcePath: "/tmp/weather/report.md", Path: "report.md"},
		{SourcePath: "/tmp/weather/summary.txt", Path: "summary.txt"},
	},
})
if err != nil {
	return err
}
_ = result.RunID
```

`PipelineID` is required and selects the configured distributor workflow for this upload. `ID` is the source manifest id and identifies the logical artifact inside that workflow. `UploadFiles` creates a temporary bundle, writes and validates a manifest, uploads the archive, and removes temporary files when the call returns. It does not write into producer source directories.

## Upload An Existing Bundle

Use `UploadBundle` when the producer already has a complete local bundle root containing `manifest.json`.

```go
result, err := client.UploadBundle(ctx, upload.UploadBundleOptions{
	PipelineID:     "weather-hourly",
	Root:           "/var/spool/weather/hourly-2026-06-07T15",
	IdempotencyKey: "weather.hourly.brentwood.20260607T150000Z",
})
if err != nil {
	return err
}
_ = result.RunID
```

`PipelineID` is required for existing bundles too. `UploadBundle` validates the local bundle by default and uploads only `manifest.json` plus manifest-listed files. Unlisted files are not uploaded.

## Result And Status

Upload success means the server returned `202 Accepted` after staging and validating the upload. It does not mean all configured destinations have published.

Poll status while the server retains the in-memory run record:

```go
status, err := client.Status(ctx, result.RunID)
if err != nil {
	return err
}
if status.Status == "failed" {
	return fmt.Errorf("distributor run failed: %s", status.Error)
}
```

Status values are `accepted`, `queued`, `running`, `succeeded`, and `failed`. Completed records expire according to `server.http.retention`; server restart clears run status and idempotency records.

## Idempotency And Retry

Every upload request includes `Idempotency-Key`.

If `IdempotencyKey` is omitted, the client generates a random 128-bit lowercase hexadecimal key for that upload operation and reuses it for retries within the same call. For cross-process retry safety, producers should pass a key derived from the producer run, such as `<bundle-id>.<run-id>`.

Do not reuse the same idempotency key for multiple distinct report generations. Reuse it only when retrying the exact same run with the same token, pipeline id, and source manifest. A repeated key with the same manifest in that scope returns the original accepted run instead of enqueueing another run; a repeated key with different content returns an idempotency conflict.

The client retries only safe cases:

- `503 Service Unavailable`;
- temporary network errors;
- ambiguous mid-upload failures.

It does not retry after `202 Accepted` and does not retry `400`, `401`, `403`, `404`, `409`, `413`, or `415`.

Detect conflicting key reuse with `errors.As`:

```go
var conflict *upload.IdempotencyConflictError
if errors.As(err, &conflict) {
	return fmt.Errorf("idempotency key was reused for different bundle content: %w", err)
}
```

## Boundaries

`pkg/upload` does not configure server pipelines, choose destinations, wait for publication completion automatically, persist client queues, provide durable idempotency across server restarts, or expose destination state. It submits complete source bundles to the configured HTTP upload API.