How Transfers Work

A Beam transfer moves data from a source to a destination by splitting it into chunks and distributing those chunks across multiple workers. This page follows a transfer from request to completion.

You create a transfer with an SDK or with a Beam Transfer step in Beam Studio. You name one or more sources and one or more destinations. For object storage, the SDK reads the source object's size, prepares the transfer, and signs short-lived, chunk-scoped access to your storage in your own process, so your storage credentials are never sent to Beam or to a worker.

transfer = await beam.transfers.create_transfer(
    sources=[S3CompatibleProviderSource(**storage, bucket="source-bucket", key="data/file.bin")],
    destinations=[
        S3CompatibleProviderDestination(**storage, bucket="destination-bucket", key="data/file.bin")
    ],
    name="dataset-transfer",
)

Creating a transfer hands it to the network straight away. The Developer Quickstart has the complete program, and Beam SDK the same call in TypeScript and Go.

There is no HTTP endpoint that creates a transfer. The BeamCore HTTP API reads transfers once they exist.

2. Assignment

Beam assigns chunks to orchestrators that compete on measured delivery performance. Each orchestrator hands its chunks to workers in its pool.

3. Worker execution

For each chunk it is given, a worker:

Reads the chunk's byte range from the source.
Writes it to the destination as one part.
Reports the chunk as done.

4. Completion

Once every chunk is verified, Beam completes the destination object and the transfer becomes completed. Your code sees it through wait_complete in Python, or the equivalent call in the other SDKs. The transfer is listed under Transfers in the Console, and the BeamCore HTTP API returns the same status.

Chunk-scoped access

For object storage, the SDK signs access per chunk. A worker receives only the short-lived source read and destination write its chunk needs, never your bucket credentials or unrestricted access to the full source or destination object. Beam checks the uploaded parts and completes the destination object after all chunks are done.

See Connectors for the permissions each provider needs.

Transfer Statuses

Status	Meaning
`pending`	Created, awaiting task assignment
`in_progress`	One or more chunks being transferred
`completed`	All chunks verified
`failed`	Terminal error — the transfer or a chunk could not complete
`cancelled`	Cancelled by client or timeout

Transfer errors

A failed transfer's error_message starts with one of these codes when your storage refuses Beam.

Code	Meaning	Check
`source_access_denied`	The source storage refused Beam.	Read permission; no IP or network restriction.
`destination_access_denied`	The destination storage refused Beam.	Write permission; no IP or network restriction.

Chunking

Files are split using BeamCore's chunking configuration. The current minimum chunk size is 40 MiB, and chunk size grows for large transfers.

Chunk boundaries are deterministic for a given file size. Each chunk is an independent unit of work that a different worker, under a different orchestrator, can execute, which is what lets a large transfer run in parallel.

Fault Tolerance

If a participant stalls or fails, Beam reassigns its chunks automatically; your transfer continues without action from you.

Transfer Lifecycle

Step-by-Step

1. Create a transfer