GraphChi: Memory budget exceeded with

User 666 | 9/3/2014, 9:25:47 PM

Hello everybody,

I am using GraphChi on my PC before moving on a cluster with GraphLab. I often obtain the "Memory budget exceeded with" notification during the computation. I guess it means that the adjacencies of a node does not fit into the memory. What does it happen in this case? Do we get an approximate result obtained by skypping some vertex's computation?

In some case when I reduce the membudget, I get the notification at every single step and the overall computation seems not to terminate (I tried big graphs with the pagerank and connectedcomponents examples).

Would you clarify this aspect to me? I could not get it from the code. Thanks

Comments

User 6 | 9/4/2014, 6:11:53 AM

Hi, GraphChi uses out of core processing, namely the data is loaded into memory if you have enough memory. Once you run out of memory the data is read from disk. This informative message let's you know once data is loaded from disk. This is not an error or warning. You can increase the memory budget using the command line flag membudget_mb=XXX to allow for larger memory buffer.

You may be interested in learning that we are working on Graphlab Create, which is our newer software which does also out of core processing. GraphLab Create engine will be release soon as an open source project. You are welcome to watch our keynote video for learning about the advanced capabilities: https://www.youtube.com/playlist?list=PLRu6_g339G-f-UuGxxPRzJeYHN6lItKCK


User 666 | 9/4/2014, 9:14:25 PM

Thanks Danny. I'm really interested in this aspect of the graph processing. Looking forward to the opensourcing of GraphLab Create.