Hadoop class path

User 647 | 8/30/2014, 12:47:08 AM


I had some issues with GraphLab Create not finding some Hadoop jar files.

__getexpandedclasspath() expands path such as /usr/lib/hadoop/lib/* but the class path can also include full paths to the .jar files, such as /a/b/c.jar

A possible fix is to replace the current list comprehension with:

<pre> jars = {path for path in classpath.split(':') if path.endswith('.jar')} for path in classpath.split(':'): if path.endswith('*'): jars.update({os.path.abspath(jarpath) for jarpath in _glob.glob(path + '.jar')})

jars = ':'.join(jars)


No Comments