Why Number of processes "hard limit" is 128?

User 293 | 5/7/2014, 3:16:23 AM

I found Line 498 in dc.hpp of GraphLab 2.2 is "ASSERTMSG(machines.size() <= RPCMAXNPROCS, "Number of processes exceeded hard limit of %d", RPCMAXNPROCS);". And RPCMAXNPROCS is defined 128 now. Is that mean the size of cluster running GraphLab concurrently is 128 at most? So far, what size is GraphLab running in industrial world? Are there any reasons for "128"? Or is "128" well tested? If we want to increase the size, are there any problems? Thanks very much!


User 20 | 5/7/2014, 6:06:34 PM

You can bump up that number. The reason is that we use a bitfield in the graph to hold information about the graph partitioning, and a constant limit (128) means that that we can hold that with less memory + no heap allocations. Bumping that up to 256 for instance, will incur an additional 16 byte overhead per vertex. Thats it.