lambda workers did not start

User 956 | 11/19/2014, 10:13:59 AM

Hello everyone,

When I execute the code from the documentation I always get the following error: <blockquote class="Quote">failed: RuntimeError: Runtime Exception. Unable to evaluate lambdas. lambda workers did not start</blockquote>

The code from the documentation <code class="CodeInline">sa = graphlab.SArray([1,2,3]) sa.apply(lambda x: x*2)</code>

I am running on a MacBook Pro Retina with OSX 10.10.1 2 GHz i7, 8gb RAM

Is there any other solution than running in a virtualenv?

Thanks for your help Tarek

Comments

User 956 | 11/19/2014, 12:48:53 PM

Sorry there war a misunderstanding of virtualenv. I just thought it would be a real virtual machine and didn't know that it's a python thing. So I guess that will solve my problem. Please excuse this post.

For all the other guys who are new to python and graphlab <a href="http://docs.python-guide.org/en/latest/dev/virtualenvs/">virtualenv</a>


User 89 | 11/21/2014, 12:19:13 AM

Hello Tarek,

There are several possible causes of this, but typically it happens when the underlying engine that spawns the lambda worker threads somehow pick up a different python version or a different graphlab version. Typically, running it in a virtualenv helps. If not, take a look at the log file where the error messages are dumped, let us know what errors you are encountering, and we'll look into it. The path to this log file is printed when you first import graphlab.

Thanks! -- Hoyt


User 942 | 11/23/2014, 1:41:34 PM

Same issue here on Mac OS 10.10.1


User 956 | 11/23/2014, 5:13:38 PM

Virtualenv solved my problem

Thanks! Tarek


User 1202 | 1/15/2015, 10:20:29 PM

I'm having this same problem. I was running graphlab inside of Anaconda but in response to this thread I created a new virtualenv (--no-site-packages) and reinstalled everything. Still having the same issue.

I was doing a few slightly non-standard things: 1) running inside an iPython notebook started by root (since I was having a crash from running out of file descriptors and wanted to be able to increase that in script) and 2) using a bind mount over /var/tmp because graphlab was filling up my root disk. But I have now tried running as a normal user, running from the command line, and undoing the bind mount so everything should be running normally.

The error is being triggered by a call to .evaluate() on a recommender model I've created.

Here's the log output: https://gist.github.com/rogueleaderr/358c6508e50932fc6c81 Here's what I have installed in my virtualenv: https://gist.github.com/ef81e82aea8b363f0c9c

Any suggestions?


User 1189 | 1/16/2015, 9:41:04 PM

Hi,

You can increase file descriptors in the current shell session with ulimit -n [bigger number]. Though, we should be quite robust to limited file descriptors now. (ever since 1.0 I think).

You can also change the cache file locations by setting the environment variable:

GRAPHLABCACHEFILE_LOCATIONS="/somewhere/else"

Or at runtime with

gl.setruntimeconfig('GRAPHLABCACHEFILE_LOCATIONS', '/somewhere/else')

You can see the options with help(gl.setruntimeconfig)

Sorry, we should be documenting these better. Not sure why they are not appearing. We will try to put these in soon.

We are working on trying to make the lambda workers more robust in 1.3. There is some issues trying to get it to pick up the correct python installation, and site-packages paths when there are multiple pythons installed.

When using conda, the most reliable thing to do is to use conda's own virtualenv system. See "Creating and Using environments" on http://www.continuum.io/blog/conda.

CONDA_PATH/bin/conda create -n [name] anaconda
CONDA_PATH/bin/activate [name] 

User 1191 | 2/2/2015, 7:32:05 PM

Hello,

I am also getting the same error. I'm using graphlab with virtualenv.

<pre class="CodeBlock"><code>sf = gl.loadsframe( pathtosframedata ); sf["INSAMPLEFOLD"] = sf_["INSAMPLE_FOLD"].apply(lambda x : int(x) if not isnan(x) else -1 )

<repr(<graphlab.data_structures.sarray.SArray at 0x57da550>) failed: RuntimeError: Runtime Exception. Unable to evaluate lambdas. lambda workers did not start></code></pre>

The details of my system are: <pre class="CodeBlock"><code>[firdaus@hs: 19:24:25] {~}[259]>python --version Python 2.7.6

[firdaus@hs: 19:31:29] {~}[262]>uname -a Linux hs.firdaus 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt2-1~bpo70+1 (2014-12-08) x86_64 GNU/Linux</code></pre>

And: <pre class="CodeBlock"><code>In [3]: gl.version Out[3]: '1.2.1.gpu'</code></pre>

Please advise on how to proceed.

Thanks !


User 1189 | 2/2/2015, 11:30:03 PM

Hi fjanoos,

You don't seem to be running Anaconda, so things should normally work... Can you try running in a virtualenv?

Essentially we need libpython2.7.so for the lambda workers. Running in a virtualenv is the easiest way to ensure that. Or alternatively:

locate libpython2.7.so
export LD_LIBRARY_PATH=[a directory containing libpython2.7.so]

User 1375 | 3/6/2015, 8:44:48 PM

Fyi, I experienced the same issue in this thread (hanging workers on an SFrame.apply operation) and decided to solve it more exotically, or rather, go around the problem. I spun up an EC2 instance with aws.launch_EC2. Any subsequent SFrame.apply operations ran like a charm (since the now remote Graphlab Server is all well set up, python-wise).


User 2402 | 10/12/2015, 6:49:22 PM

If you are running GraphLab in Linux set LDLIBRARYPATH as I did below:

export LD_LIBRARY_PATH='/usr/lib:/lib:${LD_LIBRARY_PATH}'

This should fix the issue.

Thanks to Peregin55 for exposing the issue on this thread:

http://forum.dato.com/discussion/1281/unable-to-start-graphlab-create


User 1191 | 12/17/2015, 4:32:49 AM

Hello, I am also getting the same error - even with virtualenv.

sf = gl.loadsframe( pathtosframe ); sf_["INSAMPLEFOLD"].apply( int(x) if not isnan(x) else -1 ) <repr(<graphlab.datastructures.sarray.SArray at 0x57da550>) failed: RuntimeError: Runtime Exception. Unable to evaluate lambdas. lambda workers did not start>

Below is the log for the session

mia/v8/gl2/2001/02 1422902225 : INFO: (createarraysforreading:107): Opening Frame for Reading of size (64265,388) 1422902225 : INFO: (head:411): Function entry 1422902225 : INFO: (newcache:145): Cache Utilization:0 1422902225 : INFO: (copy:103): Function entry 1422902225 : INFO: (close:505): Writing column 0 with total utilization of 8 1422902225 : INFO: (beginiterator:2504): Function entry 1422902225 : INFO: (iteratorgetnext:2518): Function entry 1422902225 : INFO: (transform:429): Function entry 1422902225 : INFO: (head:411): Function entry 1422902225 : INFO: (copy:103): Function entry 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000001 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000005 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker ...... 1422902225 : INFO: (spawnworker:51): Worker pid = 18489 1422902225 : INFO: (spawnworker:51): Worker pid = 18493 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000013 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000014 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18488 1422902225 : INFO: (spawnworker:51): Worker pid = 18491 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000012 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18492 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000000 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000002 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000011 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000014 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000000 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000004 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000015 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18495 1422902225 : INFO: (spawnworker:51): Worker pid = 18500 1422902225 : INFO: (spawnworker:51): Worker pid = 18498 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000002 usiHTTP/1.1 200 OK Transfer-Encoding: chunked Date: Thu, 21 Jul 2016 23:13:36 GMT Server: Warp/3.2.6 Content-Type: application/json

016A ["37zyefqi2sweveyp","42fn7zeo6v5ui427","66pt5sk2wz2jrbzu","awoljknjigytdyls","cj2lanoogknwopto","cnm3adnh35xmsx3f","ebxs4t2y6xr5izzy","eg5zus2pz72mr7xb","exshwew2w2jv3n7r","hxrxgzvgms3incmf","hymu5oh2f5ctk5jr","jkisbjnul226jria","lag7djeljbjng6bu","o3l65o4qzcxs327j","qsk2jzo2zh523r24","t7k6g7fkndoggutd","xfllvjyax4inadxh","ygtjzi2wkfonj3z7","yycjajwpguyno4je"] 0


User 1191 | 12/17/2015, 4:32:49 AM

Hello, I am also getting the same error - even with virtualenv.

sf = gl.loadsframe( pathtosframe ); sf_["INSAMPLEFOLD"].apply( int(x) if not isnan(x) else -1 ) <repr(<graphlab.datastructures.sarray.SArray at 0x57da550>) failed: RuntimeError: Runtime Exception. Unable to evaluate lambdas. lambda workers did not start>

Below is the log for the session

mia/v8/gl2/2001/02 1422902225 : INFO: (createarraysforreading:107): Opening Frame for Reading of size (64265,388) 1422902225 : INFO: (head:411): Function entry 1422902225 : INFO: (newcache:145): Cache Utilization:0 1422902225 : INFO: (copy:103): Function entry 1422902225 : INFO: (close:505): Writing column 0 with total utilization of 8 1422902225 : INFO: (beginiterator:2504): Function entry 1422902225 : INFO: (iteratorgetnext:2518): Function entry 1422902225 : INFO: (transform:429): Function entry 1422902225 : INFO: (head:411): Function entry 1422902225 : INFO: (copy:103): Function entry 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000001 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000005 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker ...... 1422902225 : INFO: (spawnworker:51): Worker pid = 18489 1422902225 : INFO: (spawnworker:51): Worker pid = 18493 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000013 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000014 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18488 1422902225 : INFO: (spawnworker:51): Worker pid = 18491 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000012 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18492 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000000 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000002 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000011 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000014 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000000 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000004 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:27): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000015 using binary: /srv/local/1/localinstall/lib/python2.7/site-packages/graphlab/pylambdaworker 1422902225 : INFO: (spawnworker:51): Worker pid = 18495 1422902225 : INFO: (spawnworker:51): Worker pid = 18500 1422902225 : INFO: (spawnworker:51): Worker pid = 18498 1422902225 : INFO: (spawnworker:35): Start lambda worker at ipc:///var/tmp/graphlab-firdaus/17610/000002 usiHTTP/1.1 200 OK Transfer-Encoding: chunked Date: Thu, 21 Jul 2016 23:13:36 GMT Server: Warp/3.2.6 Content-Type: application/json

016A ["37zyefqi2sweveyp","42fn7zeo6v5ui427","66pt5sk2wz2jrbzu","awoljknjigytdyls","cj2lanoogknwopto","cnm3adnh35xmsx3f","ebxs4t2y6xr5izzy","eg5zus2pz72mr7xb","exshwew2w2jv3n7r","hxrxgzvgms3incmf","hymu5oh2f5ctk5jr","jkisbjnul226jria","lag7djeljbjng6bu","o3l65o4qzcxs327j","qsk2jzo2zh523r24","t7k6g7fkndoggutd","xfllvjyax4inadxh","ygtjzi2wkfonj3z7","yycjajwpguyno4je"] 0