Very High network transfers from ONE slave only

User 11 | 10/15/2015, 12:58:48 PM

Hi all,

I have an interesting situation that I hope you can explain. I run CC and SSSP algorithms in a large graph using 64+1 machines. Each has 30 GB memory.

During the execution, I found that only one machine (the first slave machine mentioned in the hadoop/conf/slaves file) has a very high out-network transfers. I am using AWS EC2 to monitor the cluster. It sends about 2 GB every minute and receives 100 MB per minute. All other slaves send about 30MB per minute, and receive about 50 MB per minute.

It looks like PowerGraph elected a slave to do a job, and it consistently sends out this information to all slaves. Does any one know what is the reason for that ?

Thanks, -Khaled

Comments

User 1174 | 10/16/2015, 6:44:31 PM

Hello,

PowerGraph is no longer supported. Please try using GraphLab-Create!