how can pagerank in the Hadoop file system?

User 926 | 11/16/2014, 1:16:16 PM

Hi, I am new to Graphlab. Now, i have a file put it into the Hadoop file system. i want to use the graph analytic's pagerank to run this file with 1 master and 2 slaves. How can i do it? Thanks!!!!!

Comments

User 6 | 11/16/2014, 4:04:10 PM

An example of using PowerGraph here: http://graphlab.org/projects/tutorials.html#cluster

However, we strongly recommend switching to GraphLab Create, which has a pagernk implementation whcih supports HDFS: http://graphlab.com/products/create/docs/generated/graphlab.pagerank.create.html


User 926 | 11/16/2014, 6:13:37 PM

Hi Danny, Thank you. My friend also ask me to see the example in http://graphlab.org/projects/tutorials.html#cluster but i don't understand mpiexec -n 2 -hostfile ~/machines /path/to/als --matrix /some/ns/folder/smallnetflix/ --maxiter=3 --ncpus=1 --minval=1 --maxval=5 --predictions=outfile". because now i have a master and 2 slaves, and the file is located in the Hadoop file system. Where shall i put the path for the above commands? also, how can i know they are working with these 3 machines?


User 926 | 11/16/2014, 7:16:20 PM

then, i tried the command mpiexec -n 2 -hostfile ~/machines env CLASSPATH=~/hadoop/hadoop-core-1.2.1.jar ~/graphlab/release/toolkits/graph_analytics/pagerank --graph=hdfs://54.64.11.247:9000/user/ubuntu/pagerank.tsv --format=tsv --saveprefix=hdfs://54.64.11.247:9000/user/ubuntu/output

it shows the following error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:146) Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 1 more Can't construct instance of class org.apache.hadoop.conf.Configuration ERROR: hdfs.hpp(hdfs:111): Check failed: filesystem != __null

I had checked the hdfs path is correct and i don't know why?