als with HDFS Error

User 1577 | 3/19/2015, 8:27:42 AM

Hi Danny tried to run als with HDFS: /usr/mpi/gcc/openmpi-1.8.4/bin/mpiexec -n 4 -hostfile ~/machines env CLASSPATH=hadoop classpath ./als hdfs://r-aa-fatty09-b0:8020/tmp/alsexample --maxiter=3 --ncpus=1 i get the following error: <i class="Italic"><b class="Bold">INFO: distributedgraph.hpp(setingressmethod:3201): Automatically determine ingress method: grid minval: -1e+100 maxval: 1e+100 Loading graph. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) Can't construct instance of class org.apache.hadoop.conf.Configuration ERROR: hdfs.hpp(hdfs:111): Check failed: filesystem != __null Exception in thread "main" Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration java.lang.NoClassDefFoundError: org/apache/hadoop/conf/ConfigurationCaused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration

    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.security.AccessController.doPrivileged(Native Method) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)Can't construct instance of class org.apache.hadoop.conf.Configuration ERROR: hdfs.hpp(hdfs:111): Check failed: filesystem != __null

    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

Can't construct instance of class org.apache.hadoop.conf.Configuration ERROR: hdfs.hpp(hdfs:111): Check failed: filesystem != null Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) Can't construct instance of class org.apache.hadoop.conf.Configuration ERROR: hdfs.hpp(hdfs:111): Check failed: filesystem != null


mpiexec noticed that process rank 0 with PID 31617 on node r-aa-fatty09-b0 exited on signal 6 (Aborted). --------------------------------------------------------------------------</b></i>

when i remove the classpath i still get the error , just withought the java exception.

Comments

User 1592 | 3/19/2015, 8:37:14 AM

It looks like something is wrong with the classpath. Maybe hadoop is not in the path? when you run the command "hadoop classpath" in the terminal does it work? Maybe try to run /path/to/bin/hadoop classpath instead


User 1577 | 3/19/2015, 9:18:04 AM

Hi Danny i embedded the hadoop classpath inside of the als command. when i try to export it to env variable and use it as a classpath it still raise the error. <i class="Italic"><b class="Bold">[shlomi@r-aa-fatty09-b0 collaborative_filtering]$ hadoop classpath /etc/hadoop/conf:/usr/lib/hadoop/lib/:/usr/lib/hadoop/.//:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/:/usr/lib/hadoop-hdfs/.//:/usr/lib/hadoop-yarn/lib/:/usr/lib/hadoop-yarn/.//:/usr/lib/hadoop-mapreduce/lib/:/usr/lib/hadoop-mapreduce/.// [shlomi@r-aa-fatty09-b0 collaborative_filtering]$</b></i> Also /usr/bin/hadoop classpath is the same


User 1592 | 3/31/2015, 6:54:52 AM

It may still mean that your classpath is wrong.. especially check the "*" signs