Read and write to HDFS

User 459 | 7/10/2014, 11:40:05 PM

I have compile toolkit examples and try to read data from HDFS but I am getting following message on every example I have compiled from graph lab source

FATAL: distributedgraph.hpp(loadfrom_hdfs:2226): Attempting to load a graph from HDFS but GraphLab was built without HDFS.

mpiexec -n 2 env CLASSPATH=hadoop classpath graphlabprogram

Even passing hadoop classpath is not working. How do you build graph lab with HDFS? Need suggestions please.


User 459 | 7/11/2014, 5:35:18 PM

Any generous people willing to help me here? I have been trying to fix this issue for the past few days but no luck. Not even any suggestion from anyone. At least I was hoping some suggestions from Graphlab team :(

User 459 | 7/11/2014, 9:01:01 PM

After pulling my hair for 3 days I finally knew what was the problem. In the graphlab home folder CMakeLists.txt file checks for JAVAHOME folder and look for jni.h. It load libhdfs only if it finds jni.h. Then I realize that I have not set JAVAHOME folder. This was the reason why i was getting a message "Graphlab was not built with HDFS" . After setting JAVA_HOME folder I was able to run program reading and writing data from HDFS.

One more thing, if you are using hamster but not openmpi then in in order to run the program you have to export HADOOP_CLASS path else it will give class not found exception.

Finally I am a happy man !!