Deep Learning in Graphlab

User 67 | 3/25/2014, 10:26:11 AM

Hi,

How about deep learning in Graphlab? Are there any plans in near future to implement deep learning algorithms in Graphlab? Jeffrey Dean in his paper says that "GraphLab, designed for general (unstructured) graph computations, would not exploit computing efficiencies available in the structured graphs typically found in deep networks." May I know about this comment in the paper in detail?

Comments

User 67 | 3/25/2014, 10:47:54 AM

I think Graphlab has computing efficiencies of Structured data as well.


User 18 | 3/25/2014, 5:26:58 PM

Hi,

Jeff's comment in his NIPS 2012 paper was meant to point out that the general formulation of GraphLab's graph computation model does not take advantage of deep learning network's specific architecture. It's a correct statement, but it does not preclude the fact that we can tailor the computation model to deep learning in the future. We plan to support a wide array of iterative machine learning computations, including deep learning.

Thanks, Alice


User 67 | 4/23/2014, 10:37:57 AM

Hi,

In order to investigate the claim of Jeffrey Dean about the graphlab W.R.T structured graphs in his paper, I coded a small deep neural network (10 hidden layers with each layer having 10 nodes). I used Contrastive Divergence (CD) algorithm for learning the structure in data. I am facing problem in Scatter phase, however I think I may resolve that. I implemented Run time scheduling feature to communicate between the layers. One of the problems I am facing is the parallelism across the data set. I mean if there are 1 million entries, it would be good if I could be able to divide the dataset into 10 different sets and I can pass each dataset to typical graphlab engine and computing each engine in parallel. Is it possible to do so?

One more feedback I would like to share is how about signalling a layer instead of signalling a particular vertex in the layer? After the structure of graph is created, I am loading the graph for every dataset from the hard disk and then re-computing the structure of graph and then writing back to the disk so that next data set could pick up the changed graph. It seems there is lot of I/O in this process. Is there any method to by-pass this IO?

It would be great if you feed back about the changes that could be made to our Graphlab to suit structured graphs model. It would be really interesting for me to explore this problem in depth.

Thanks, Sagar.