Out of memory on my slave

User 872 | 12/5/2014, 12:30:32 PM

Hi,

When I load an adj format graph with powergraph, my slave shows out of memory after 171638988 lines. Besides, the master shows FATAL: dctcpcomm.cpp(sendtillblock:311): send error: Connection reset by peer. My graph file is 236GB and my memory in the two nodes are both 128GB. Does it mean that my file is too large? And would some one tell me how to solve this problem?

Best, Lyuwei

Comments

User 14 | 12/5/2014, 6:39:40 PM

You will want more nodes. Also, if you do not need dynamic graph, you can save some memory by removing "adddefinition(USEDYNAMICLOCALGRAPH" in the graphlab/CMakeLists.txt


User 872 | 12/7/2014, 8:09:55 AM

Thank you JayGu. However this does not works. May there be other solutions? Thank you very much.


User 6 | 12/7/2014, 9:37:11 AM

Which algorithm are you using ? We would love helping you utilizing Graphlab Create on a single machine and verify it works for your input graph.


User 872 | 12/7/2014, 10:45:31 AM

Hi, Danny. I use pagerank, but it crash at loading data. Can Graphlab create run on multi-nodes now? If so, I will glad to use it! Thank you very much!


User 6 | 12/7/2014, 11:56:02 AM

GraphLab Create runs on a single node but can scale to much bigger problems than GraphLab on a single node, because we do not load the problem fully into memory. We map the input from disk. <a href="http://graphlab.com/products/create/docs/generated/graphlab.pagerank.create.html#graphlab.pagerank.create">Here</a> is an example for running pagerank. Please try it out and ping us for any issues.


User 872 | 12/7/2014, 1:22:33 PM

Thank you, Danny. However my tutor asks me to use multi-nodes system, maybe I have to use powergraph now. So would you tell me the method GraphLab Create use to load graph partly into memory. Maybe I should modify powergraph code by myself if I cannot get more nodes. Thank you very much!


User 6 | 12/7/2014, 2:08:17 PM

We are working on a future version of Graphlab Create that will be distributed.

In the meantime, I suggest that you verify you have disabled the dynamic graph, and then reconfigure and recompile.

It is also recommended to split the input file into disjoint parts.


User 872 | 12/8/2014, 1:27:52 AM

Thank you Danny. I have disabled dynamic graph as JayGu said. So do you mean I should part my input file like input.part1, input.part2?


User 872 | 12/8/2014, 4:27:23 AM

I split the input into input.part1, input.part2, input.part3, however powergraph seemed to run them at the same time so the whole file still cannot be read into memory.