User 11 | 10/15/2015, 12:58:48 PM
I have an interesting situation that I hope you can explain. I run CC and SSSP algorithms in a large graph using 64+1 machines. Each has 30 GB memory.
During the execution, I found that only one machine (the first slave machine mentioned in the hadoop/conf/slaves file) has a very high out-network transfers. I am using AWS EC2 to monitor the cluster. It sends about 2 GB every minute and receives 100 MB per minute. All other slaves send about 30MB per minute, and receive about 50 MB per minute.
It looks like PowerGraph elected a slave to do a job, and it consistently sends out this information to all slaves. Does any one know what is the reason for that ?