distributor/docs/internal/ingest.md

# Ingestion Internals

Audience: developers and LLM coding agents changing `internal/ingest`.

## Purpose

`internal/ingest` validates upload content types, extracts uploaded source bundle archives into local temporary storage, validates extracted bundles, and commits accepted bundles to per-run staging directories.

## Inputs And Outputs

Inputs are a context, upload body reader, content type, pipeline staging path, run id, maximum uploaded size, maximum extracted size, and maximum file count. Output is a `StagedBundle` containing the committed local bundle root and parsed manifest.

## Boundaries

The package does not authenticate HTTP requests, manage upload queues, track upload status, publish destinations, load config, or start an HTTP server. Those responsibilities live in `internal/app`.

The HTTP API contract is documented in `docs/integrations/http-upload.md`.

## Config Fields Used

The package does not read config directly. The app layer passes effective values derived from `source.staging_path`, `source.max_upload_size`, and HTTP server defaults.

## Adapters Used

The package uses the local filesystem directly for temporary archive storage, extraction, validation, and final staging path promotion. It does not use the storage backend abstraction.

## State And Manifest Behavior

Accepted archives must contain exactly one root-level `manifest.json`. After extraction, the package validates the staged root through `pkg/bundle`, including manifest parsing, source path rules, file existence, regular-file checks, file sizes, file SHA-256 digests, and bundle digest.

## Skip And Resume Behavior

The package has no resume behavior. A successful call commits one complete staged bundle root. Failed calls remove temporary data created by that call.

## Failure Behavior

Failures include unsupported content type, unsafe run id, missing staging path, non-positive limits, oversize upload body, oversize extracted content, too many files, unsafe archive paths, duplicate files, nested manifests, unsupported tar entry types, gzip/tar read errors, bundle validation errors, and filesystem errors.

## Tests To Inspect

- `internal/ingest/archive_test.go`
- `internal/app/upload_*_test.go`
- `pkg/bundle/*_test.go`

## Architectural Invariants

- Invalid archives never commit a staged root.
- Archive paths remain clean relative slash-separated paths.
- Only directories and regular files are accepted from tar archives.
- Source bundle validation happens before final staging path promotion.
- Upload authentication and queueing remain outside this package.