GraphLab Create. Using Anaconda ipython-notebook. Lambda workers did not start

User 2710 | 12/1/2015, 8:29:34 PM

Hello,

I tried the following code in my Anaconda ipython-notebook:

def transform_country(country): if country == 'USA': return 'United States' else: return country

sf['Country'].apply(transform_country)


I then receive the following error:


RuntimeError Traceback (most recent call last) <ipython-input-23-58b2d3ea026a> in <module>() ----> 1 sf['Country'].apply(transform_country)

C:\Users\spiegelc\AppData\Local\Continuum\Anaconda\lib\site-packages\graphlab\datastructures\sarray.pyc in apply(self, fn, dtype, skipundefined, seed) 1691 1692 with cythoncontext(): -> 1693 return SArray(proxy=self.proxy.transform(fn, dtype, skipundefined, seed)) 1694 1695

C:\Users\spiegelc\AppData\Local\Continuum\Anaconda\lib\site-packages\graphlab\cython\context.pyc in exit(self, exctype, excvalue, traceback) 47 if not self.showcythontrace: 48 # To hide cython trace, we re-raise from here ---> 49 raise exctype(exc_value) 50 else: 51 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start.


I installed python via anaconda. My python version is 2.7.9-1. This has been done using the ipython-notebook. Any help would be greatly appreciated.

Comments

User 954 | 12/1/2015, 9:09:25 PM

Hello,

Can you let us know the GLC version and operating system you are using?

Thanks.


User 2709 | 12/1/2015, 10:01:10 PM

Hi Soroush,

I have the same problem. Same as Chris, I am pursuing the Machine Learning Specialization on Coursera. I think I have downloaded the latest GLC version via Dato Launcher. I am using Windows 64 bit OS, 16 GB RAM


User 1207 | 12/2/2015, 1:39:44 AM

Hello Chris and Nimahtehrani,

Could you run the following commands in the notebook and paste the output here? It's going to spit out a lot of diagnostic information, but it should help us track down the problem.

import graphlab print graphlab.version graphlab.sys_util.test_pylambda_worker()

Thanks! -- Hoyt


User 2710 | 12/2/2015, 1:22:46 PM

Soroush and Hoytak,

I am using Windows 7 Enterprise. 8gb ram. 64-bit OS.

The Graph Lab version is 1.7.1.

When I run the code Hoytak provided, the output is: 1.7.1

Launch pylambdaworker process with simulated unityserver environment.

Thanks again!

Chris


User 2716 | 12/2/2015, 3:31:15 PM

Hi hoytak, I have the same issue. After running your test codes, my windows 7 with 12g memory, prompts the following picture ~~~~~~~~~update~~~~~~~~~~ surprised, when I run the code 2nd , it displays the following info 1.7.1

Launch pylambdaworker process with simulated unityserver environment. but apply still does not work for me


User 15 | 12/2/2015, 7:49:47 PM

@gamesover Do you have a file somewhere in your system called libeay32.dll? This is a dll that we package and it should be fully compatible with the lambda worker dll, but you may have a libeay32.dll somewhere on your system that it's finding first, and that one may be an older (or newer) version than what we expect. Can you try to find this file?


User 1207 | 12/2/2015, 8:00:12 PM

Thanks, appreciate it!

We are trying to track this down on our end. One more thing (that I forgot to specify). When running the last command in the ipython notebook, it seems that all the diagnostic information is actually spit out to the terminal and not displayed in the ipython notebook itself. Would it be possible to run the above commands in either ipython directly or to copy the diagnostic information from the terminal? What I get starts with

PyLambda script called with no IPC information; entering diagnostic mode. Script directory: /home/hoytak/test/gl-test/local/lib/python2.7/site-packages/graphlab/_pylambda_worker.pyc. Main program directory: /home/hoytak/test/gl-test/local/lib/python2.7/site-packages/graphlab. Found 1 candidade pylambda_worker file(s): /home/hoytak/test/gl-test/local/lib/python2.7/site-packages/graphlab/libpylambda_worker.so. Installation Directory Structure: <root>//__init__.py: posix.stat_result(st_mode=33188, st_ino=3015222, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=9373, st_atime=1447957954, st_mtime=1447957949, st_ctime=1447957953) <root>//__init__.pyc: posix.stat_result(st_mode=33188, st_ino=3015482, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=9326, st_atime=1449020149, st_mtime=1447957954, st_ctime=1447957954) <root>//_distributed_execution_environment.py: posix.stat_result(st_mode=33188, st_ino=3015224, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=3999, st_atime=1447957954, st_mtime=1447957949, st_ctime=1447957953) <root>//_distributed_execution_environment.pyc: posix.stat_result(st_mode=33188, st_ino=3015484, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=4346, st_atime=1449020149, st_mtime=1447957954, st_ctime=1447957954) <root>//_gl_pickle.py: posix.stat_result(st_mode=33188, st_ino=3015221, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=18345, st_atime=1447957954, st_mtime=1447957949, st_ctime=1447957953) <root>//_gl_pickle.pyc: posix.stat_result(st_mode=33188, st_ino=3015481, st_dev=2085L, st_nlink=1, st_uid=1000, st_gid=1000, st_size=14727, st_atime=1449020149, st_mtime=1447957954, st_ctime=1447957954) ...

and then ends with

` 1449085375 : INFO: (pylambdaworkermain:63): Python initialized successfully. 1449085375 : INFO: (pylambdaworkermain:75): No valid server address, exiting. Example: ipc:///tmp/pylambdaworker Example: tcp://127.0.0.1:10020 Example: tcp://*:10020 Example: tcp://127.0.0.1:10020 tcp://127.0.0.1:10021 Example: ipc:///tmp/unitytestserver --authtoken=secretkey Example: ipc:///tmp/unitytestserver ipc:///tmp/unity_status secretkey

Process exited with code 1. `

There is a lot of diagnostic info about the installation in that that will hopefully allow us to pinpoint the problem. Feel free to attach it as a text file as well.

Thanks! -- Hoyt


User 2710 | 12/2/2015, 8:19:28 PM

Hoyt,

I attached the text file that contains the output from the terminal.

-Chris


User 1207 | 12/2/2015, 8:52:05 PM

Hi Chris,

Thank you. I think we have some ideas what is going wrong, but could you provide one more piece of information? When you start graphlab, it prints a log file path which has more information about the server. Could you attach that file as well?

Thanks! -- Hoyt


User 2710 | 12/2/2015, 9:15:53 PM

Hi Hoyt,

Sorry about that. I think I have it all in this txt file.

-Chris


User 1207 | 12/2/2015, 9:41:18 PM

Hi Chris,

Thanks! Sorry about the confusion -- you supplied the correct terminal output. The log file is a separate file printed by the server; if you run

import graphlab as gl gl.SArray(range(1000)).apply(lambda x:x) It should print out a line that looks something like:

[INFO] Start server at: ipc:///tmp/graphlab_server-26954 - Server binary: /mnt/workspace/bm-test-venve/lib/python2.7/site-packages/graphlab/unity_server - Server log: /tmp/graphlab_server_1449092241.log The server log file, printed last on this line, is what I'm interested in examining. Could you run the above commands and then attach that file?

Thanks so much for your patience and help; we're working hard to get to the root of this.

-- Hoyt


User 2716 | 12/3/2015, 11:11:44 AM

Hi @EvanSamanas , I found a bundle of files on my pc after search, what can I do for that? Also I attached my coursera course requires me to use apply as well.

search on my pc

original error


User 2710 | 12/3/2015, 11:49:55 AM

Hoyt,

Not a problem. The output is below:

[INFO] 1449143240 : INFO:     (initialize_globals_from_environment:282): Setting configuration variable GRAPHLAB_FILEIO_ALTERNATIVE_SSL_CERT_FILE to C:\Users\spiegel_c\AppData\Local\Continuum\Anaconda\lib\site-packages\certifi\cacert.pem
1449143240 : INFO:     (initialize_globals_from_environment:282): Setting configuration variable GRAPHLAB_FILEIO_ALTERNATIVE_SSL_CERT_DIR to 
This non-commercial license of GraphLab Create is assigned to MY-EMAIL and will expire on October 04, 2016. For commercial licensing options, visit https://dato.com/buy/.

[INFO] Start server at: ipc:///tmp/graphlab_server-7608 - Server binary: C:\Users\spiegel_c\AppData\Local\Continuum\Anaconda\lib\site-packages\graphlab\unity_server.exe - Server log: C:\Users\SPIEGE~1\AppData\Local\Temp\1\graphlab_server_1449143240.log.0
[INFO] GraphLab Server Version: 1.7.1
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-6c1ce6ff9789> in <module>()
      1 import graphlab as gl
----> 2 gl.SArray(range(1000)).apply(lambda x:x)

C:\Users\spiegel_c\AppData\Local\Continuum\Anaconda\lib\site-packages\graphlab\data_structures\sarray.pyc in apply(self, fn, dtype, skip_undefined, seed)
   1691 
   1692         with cython_context():
-> 1693             return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_undefined, seed))
   1694 
   1695 

C:\Users\spiegel_c\AppData\Local\Continuum\Anaconda\lib\site-packages\graphlab\cython\context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start.

`-Chris


User 1207 | 12/3/2015, 4:41:09 PM

Hey Chris,

In your above output, the server log file is given by Server log: C:\Users\SPIEGE~1\AppData\Local\Temp\1\graphlab_server_1449143240.log.0 This is printed in the line that begins with [INFO] Start server at: It may not be there still, but could you run the above commands again and then attach that log file?

Thanks! -- Hoyt


User 1207 | 12/3/2015, 5:01:10 PM

Hi gamesover,

The problem appears to be the old version of libeay32.dll that is stuck in your C:\system32 folder. On windows, libraries in this folder take precedence over other libraries in the system path, and most applications don't put things there for this reason.

The best solution to this problem is to reinstall the system openssl to a current version, then copy the libeay32.dll and ssleay32.dll libraries into the C:\system32 folder (in some openssl installers, this is done by default). You can get just copy these libraries from the graphlab folders -- we ship with the most recent version -- or install them from an installer from https://wiki.openssl.org/index.php/Binaries (specifically https://indy.fulgan.com/SSL/openssl-1.0.2d-i386-win32.zip has them). Make sure that in the end, the up-to-date versions are in C:\system32.

Thanks! -- Hoyt


User 2716 | 12/4/2015, 11:46:12 AM

Hi @hoytak , thanks for your reminder. I replaced c:\windows\libeay.32.dll with latest version(copy from dato folder). Issue solved. Thanks so much for your help. Just curious, in your software, there is no way to call .dll from your own installer folder?


User 2710 | 12/4/2015, 2:40:02 PM

Hoyt,

When I try to attach the log, dato says file is not allowed.

Chris


User 2679 | 12/6/2015, 6:06:13 PM

Dear @hoytak and @gamesover,

Thank your for the vivid discussion. I'm also facing this issue (also on the Coursera course).

However, when I try to cope the libeay32.dll, Windows throws an error stating "The action can't be completed because the file or folder is open in another program". In reaction, I restarted my laptop and tried again. The same error returns.

Also the application file in the .zip also throws an error, stating "can't open config file: /usr/local/ssl.openssl.cnf

Could you help me on this one? I'd be thankful


User 2296 | 12/6/2015, 9:57:44 PM

Hi, all!

I've got the same issue. Having a little deja vu -- weren't we here back in September? : D

Unfortunately, in my case, it's not related to having a libeay23.dll in C:\Windows\System32, as I don't have any such file. I found one copy of the file inside the Netbeans installation directory. I don't use OpenSSL on Windows and have never installed it directly -- I use PuTTy and its tools for generating keys, etc. And Windows has its own SSL implementation. (Which was not susceptible to Heartbleed. So there, nyah, Windows dissers!)

I have GLC 1.7.1, not using Anaconda, not using Dato Launcher -- directly installed in a virtualenv. Yes, I have a space in my user path, but I have a junction to bypass it. I started jupyter notebook with the cwd set to that path without spaces. But I see (from the output of testpylambdaworker) that there is a space in one element of the path that the pylambda worker is using. I get the same message "No valid server address, exiting." at the end. Unfortunately, it doesn't seem to be telling what server address it actually got, that it doesn't like.

I'm attaching the terminal output from running testpylambdaworker and the GLC server log listed in its startup message.

Let me know if you'd like me to try anything.

-- Pat


User 1207 | 12/7/2015, 11:33:45 PM

Hey Everyone,

Sorry for the problems. We're working on a fix for these issues. Thanks for your patience.

-- Hoyt


User 1207 | 12/8/2015, 2:17:58 AM

Hello Everyone,

So we believe we have two possible fixes for the issues above, and we would appreciate your feedback on whether they resolve the problems. If so, we will incorporate them into our next release.

These are windows-only fixes. If you are using a different OS, please let me know if you still have problems.

The first fix is a per-session fix. After starting ipython or ipython notebook in the Dato/Graphlab environment, but before importing graphlab, copy and run the following code:

` import ctypes, inspect, os, graphlab from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', uselasterror=True) kernel32.SetDllDirectoryW.argtypes = (wintypes.LPCWSTR,) srcdir = os.path.split(inspect.getfile(graphlab))[0] kernel32.SetDllDirectoryW(srcdir)

Should work

graphlab.SArray(range(1000)).apply(lambda x: x) `

The second fix is more permanent, but use this fix ONLY if you have installed graphlab using the Dato launcher and have not created your Anaconda environment. Upgrading to future graphlab versions may require an uninstall and reinstall using this method as it elevates the windows load priority of some of the graphlab libraries by copying them into the python directory. However, it is easily undone by deleting the copied libraries.

To perform this fix, copy and paste the following code into your Dato ipython or ipython notebook session, before starting graphlab:

`

Second fix; use only if using dato-launcher and no existing conda environment.

import sys, os, glob, inspect, graphlab, shutil srcdir = os.path.split(inspect.getfile(graphlab))[0] destdir = os.path.split(sys.executable)[0] alldlls = glob.glob(os.path.join(srcdir, "*.dll")) _=[shutil.copy(dll, destdir) for dll in alldlls]

Should work

graphlab.SArray(range(1000)).apply(lambda x: x) `

Thanks for your patience, everyone! -- Hoyt


User 2772 | 12/9/2015, 7:05:57 PM

I am also doing the course and have the same problem. I am using a laptop with win 7 64 bit , i3-3110m and 8gb ram and have installed graphlab through conda, so I am using the dato-env. I tried only the first solution, sadly, with no luck:

import ctypes, inspect, os, graphlab
from ctypes import wintypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
kernel32.SetDllDirectoryW.argtypes = (wintypes.LPCWSTR,)
src_dir = os.path.split(inspect.getfile(graphlab))[0]
kernel32.SetDllDirectoryW(src_dir)

graphlab.SArray(range(1000)).apply(lambda x: x)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-8020995de9bd> in <module>()
      7 
      8 # Should work
----> 9 graphlab.SArray(range(1000)).apply(lambda x: x)

D:\Anaconda2\envs\dato-env\lib\site-packages\graphlab\data_structures\sarray.pyc in apply(self, fn, dtype, skip_undefined, seed)
   1691 
   1692         with cython_context():
-> 1693             return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_undefined, seed))
   1694 
   1695 

D:\Anaconda2\envs\dato-env\lib\site-packages\graphlab\cython\context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start.

I tried adding the libeay23.dll in C:/System32 but that didnt work as well.


User 2296 | 12/9/2015, 8:38:56 PM

Don't hit me, but...may I suggest something to try? What we see in the log is this:

?[0m1449437123 : INFO:     (_pylambda_worker_main:75): No valid server address, exiting.
?[0m   Example: ipc:///tmp/pylambda_worker
?[0m   Example: tcp://127.0.0.1:10020
?[0m   Example: tcp://*:10020
?[0m   Example: tcp://127.0.0.1:10020 tcp://127.0.0.1:10021
?[0m   Example: ipc:///tmp/unity_test_server --auth_token=secretkey
?[0m   Example: ipc:///tmp/unity_test_server ipc:///tmp/unity_status secretkey

That is, the pylambda worker says that it does not have a "valid server address" and that it is exiting because it has no valid server address. It then prints what appears to be a usage statement. It is telling us what the valid forms of server address are. As I said above, what it did not print, and what seems to me to be an important clue, is the actual server address that it was given. If we know what it was given, then we might see what was wrong with that server address, and be able to change it in whatever provides the server address.

So if the Dato folks could add a print statement to print the actual server address, as well as the usage statement, then those of us who still have the problem (who are not using Anaconda, and who do not have any spurious SSL library) could run that version, and tell you what the bad server address is. Having such a print statement seems reasonable to include in the live version, so this doesn't need to be any special one-off version.

Just sayin...


User 1207 | 12/9/2015, 9:03:48 PM

@ptressel -- I agree that could be more clearly stated in the message. The diagnostic script doesn't have the server address, as the server is not started at this point; otherwise it would be printed there. It simulates the configuration and environment that the server would use to start the pylambda workers and turns on debug flags, printing out any possible issues encountered until it actually needs information from the (non-existent) server. When the lambda worker code is run normally, the server addresses are dumped to the log file.

Also, did either of the above fixes help with your problem? If not, could you try one thing. Could you replace the sysutil.py file in the graphlab python directory -- which you can determine from inspect.getfile(graphlab) -- and replace it with this file: https://raw.githubusercontent.com/hoytak/SFrame/pylambdaworkerlinuxlibraryfix/osssrc/unity/python/sframe/sys_util.py. I am wondering if a fix I put in there helps (it fixes the case I was able to repro).

Thanks so much! -- Hoyt


User 1207 | 12/9/2015, 9:15:55 PM

@Peterrr -- did you run that command from a new python session? It needs to be run before graphlab is used.

If so, could you send me the server log file? When you first start graphlab and run a graphlab command, it prints out a line containing the server address, looking something like Server log: C:\Users\<name>\AppData\Local\Temp\1\graphlab_server_1449143240.log.0 Could you send me that file? Feel free to email it to support@dato.com and reference this issue, and we'll go from there.

Thanks! -- Hoyt


User 15 | 12/11/2015, 12:01:40 AM

Hi @ptressel @Peterrr @chrisbspiegel , and whomever else has this issue and whose testpylambdaworker output returned 1,

Could I get you to gather a little more data for me?

It's clear since your diagnostic script ran successfully that the workers probably don't have conflicting library problems.

Here's what I'd like you to do: Bring up your task manager and sort the list of processes by name. Watch how many Python processes there are as you start up GraphLab Create and then run apply on an SArray or SFrame. Once you run apply how many python processes are there? Is there just one the whole time? Does the number of python processes jump to 9(ish) and quickly go back to one? Do they greatly increase in number and stay around for awhile?

Thank you so much if you are able to do this.

Evan


User 2382 | 12/11/2015, 12:04:38 AM

I am also in the Coursera course and having the same problems as described herein.

Tried to resolve via the libeay32.dll and ssleay32.dll fix described above - didn't work.

Seeing the same errors as ptressel posted in my graphlab_server.log file:

1449779813 : ERROR: (spawn_worker:82): Fail connecting to worker at ipc://C:/Users/me/AppData/Local/Temp/graphlab-me/6340/695a01c5-7579-4dee-8294-7891aab321f2. Status: Communication Failure. Retry: 1 1449779813 : ERROR: (spawn_worker:65): Lambda worker process terminated unexpectedly 1449779813 : ERROR: (operator():170): Fail launching lambda worker.

When I run the purposed first per-session fix I'm still seeing the same error:

`python Python 2.7.11 |Anaconda 2.4.1 (64-bit)| (default, Dec 7 2015, 17:02:42) [MSC v.1500 64 bit (AMD64)] Type "copyright", "credits" or "license" for more information.

IPython 4.0.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. %guiref -> A brief reference about the graphical user interface.

import ctypes, inspect, os, graphlab from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', uselasterror=True) kernel32.SetDllDirectoryW.argtypes = (wintypes.LPCWSTR,) srcdir = os.path.split(inspect.getfile(graphlab))[0] kernel32.SetDllDirectoryW(srcdir)

Out[1]: 1

graphlab.SArray(range(1000)).apply(lambda x: x) [INFO] 1449781111 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to C:\Anaconda2\lib\site-packages\certifi\cacert.pem 1449781111 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTDIR to This non-commercial license of GraphLab Create is assigned to MY@EMAIL and will expire on October 08, 2016. For commercial licensing options, visit https://dato.com/buy/.

[INFO] Start server at: ipc:///tmp/graphlabserver-1956 - Server binary: C:\Anaconda2\lib\site-packages\graphlab\unityserver.exe - Server log: C:\Users\me\AppData\Local\Temp\graphlabserver1449781111.log.0 [INFO] GraphLab Server Version: 1.7.1 Traceback (most recent call last):

File "<ipython-input-2-858d432a1322>", line 1, in <module> graphlab.SArray(range(1000)).apply(lambda x: x)

File "C:\Anaconda2\lib\site-packages\graphlab\datastructures\sarray.py", line 1693, in apply return SArray(proxy=self.proxy.transform(fn, dtype, skip_undefined, seed))

File "C:\Anaconda2\lib\site-packages\graphlab\cython\context.py", line 49, in exit raise exctype(excvalue)

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start. `

Any next steps to try? Thanks, -Myron


User 15 | 12/11/2015, 12:14:51 AM

Hi @myron,

Thanks for posting. Could you follow the instructions I posted just before you and report back?

Evan


User 2296 | 12/11/2015, 6:33:52 AM

Evan, Hoyt, et al. --

Huh. apply() just started working. Nothing has changed that I know of related to GLC or Python.

MS did the usual patch Tuesday ritual (and I again fended off the impending Win10 upgrade several times...) but it did not sneak in and install any libeay32.dll or ssleay32.dll. I cleaned up disk space but the disk had enough space available before that, and I'm fairly sure apply() failed after that. Assorted other packages snuck in and updated themselves: MySQL, McAfee, Chrome. Date on the pylambda_worker.pyc file is as expected -- last re-install of GLC, so no change there.

I tried to re-create any memory contention issues -- started Firefox, Eclipse, Chrome, Cygwin, another Python process in cmd. apply() still working. McAfee firewall and real-time scan are on.

About the number of Python processes (which may be moot since I can't repro the apply() problem just now...). When I start jupyter notebook, it launches two Python processes. Starting a notebook gets one more. Running apply gets 4 Python processes running pylambda_worker.pyc. The command lines they are launched with look like this:

c:\dato-env\scripts\python.exe "c:\dato-env\lib\site-packages\graphlab_pylambda_worker.pyc" "ipc://C:/Users/PATTRE~1/AppData/Local/Temp/graphlab-Pat Tressel/8808/..."

where ... is a different string for each, looks like a uuid or hash. The 8808 folder is currently empty.

Unfortunately, I don't have any samples of the command line or process info from a failed apply() call.

Is testpylambdaworker supposed to exit successfully, or is that error exit with "No valid server address" expected? It's still doing the same now, while apply() is working. So cancel the above suggestion. Only...is there a way to turn on debug messages from the pylambda_worker processes?


User 15 | 12/11/2015, 6:48:45 AM

Hey @ptressel,

Good grief. That's weird. I can only hope a mysterious Windows Update fixes this issue, but somehow I doubt it :).

The exit with "No valid server address" is expected. We should clean that test function up perhaps. Especially all that stupid output at the start...I'm pretty sure it's meaningless.

There isn't a way to turn on debug messages in the workers short of recompiling the libpylambda_worker.dll. I suppose you could do that seeing it's open source, but if it came down to wanting to see that, I'd probably do the leg work and send out something for people to try (and then make it easier to do it at runtime, and keep improving, and keep improving...). Getting a build environment to work on Windows on our codebase is no trivial task.

Thanks for the update!


User 2382 | 12/11/2015, 11:59:44 AM

Hi Evan,

With task manager up and tasks sorted by name (watching/sorted for python.exe) - the number of python processes does jump to 10 and then quickly goes back to two (in my case I am running IPython in a Spyder notebook). Also, tried with just a python prompt in a CMD window with the same results:


User 2772 | 12/11/2015, 12:39:59 PM

Sadly, the windows update didn't fix it for me.

@EvanSamanas

the left is beforehand, the right is after I run the first fix code snippet.

The python processes hang around after the lambda exception is thrown.


User 15 | 12/11/2015, 6:30:15 PM

Thanks to both of you for doing this!

So @myron, the processes quickly go back to 2, but they stick around long enough for you to take a screenshot? Can you estimate roughly how long they stay around?

It's somewhat troubling that @Peterrr 's processes stick around and yours don't.


User 2794 | 12/11/2015, 7:46:23 PM

Hi everybody, I'm on Mac OS 10.11.2, 4GB RAM, having similar problem with lambdas on SFrames. I've got Python 2.7.10, running it in iPython Notebook, GraphLab 1.7.1 but my sframe module runs GraphLab 1.6.

Here's a simple bit of code that gives me the lambda workers error:

import sframe
newFrame = sframe.SFrame([1,2,3])
newFrame['X1'].apply(lambda x: x**2)

But changing to:

import graphlab
newFrame = graphlab.SFrame([1,2,3])
newFrame['X1'].apply(lambda x: x**2)

makes everything work great.

Anybody know how to upgrade sframe to use graphlab 1.7.1?


User 15 | 12/11/2015, 9:17:01 PM

Hi @hertopnerd,

We did not officially release SFrame 1.7.

I know it's a little confusing, but its actually GraphLab Create that relies on SFrame, and not the other way around. To get a version of SFrame that's in line with GLC 1.7, you could build it with a branch that has HEAD set at the day we released 1.7.1, but there's no real reason that you'd want it to be "in line" with GLC version numbers. You could just build from HEAD and the problem will be fixed.


User 2382 | 12/11/2015, 9:57:14 PM

Hey Evan,

Had to be quick to capture the screen shot - the processes stick around for about three to four seconds. The .apply() function does seem to hang for about 30, or so, seconds after which the trace and RuntimeError appear.

@@hertopnerd, tried your workable solution, but still broken on my system :neutral:: `python import graphlab newFrame = graphlab.SFrame([1,2,3]) newFrame['X1'].apply(lambda x: x**2)

Traceback (most recent call last):

File "<ipython-input-10-8d31bcc38784>", line 3, in <module> newFrame['X1'].apply(lambda x: x**2)

File "C:\Anaconda2\lib\site-packages\graphlab\datastructures\sarray.py", line 1693, in apply return SArray(proxy=self.proxy.transform(fn, dtype, skip_undefined, seed))

File "C:\Anaconda2\lib\site-packages\graphlab\cython\context.py", line 49, in exit raise exctype(excvalue)

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start. `

Let me know what else to try/help with...

Thanks, -Myron


User 2794 | 12/12/2015, 7:07:59 PM

@EvanSamanas I understand this is the GraphLab support forum and note the sframe support forum, and I have to admit the build process is a little above my pay grade. :) Feel free to send me elsewhere if my questions are not appropriate for this forum.

Do I understand correctly that one day the SFrames (outside of GLC) will support .apply() someday? Perhaps GLC's SFrames are slightly more sophisticated/developed than sframe's SFrames, only because the most recent build of sframe is not released yet?


User 1207 | 12/13/2015, 1:47:56 AM

Hi @ptressel,

When your thing started working, did you try applying the earlier fix of replacing the sys_util.py file with the new one? If so, I am wondering if that did indeed fix things, but due to windows caching stuff it didn't actually show up until after your update.

To clarify for everyone else, the fix in question is to replace the sysutil.py file in the graphlab directory with a patched version available at https://raw.githubusercontent.com/hoytak/SFrame/pylambdaworkerlinuxlibraryfix/osssrc/unity/python/sframe/sys_util.py.

To do this, run

import inspect
import graphlab.sys_util
inspect.getfile(graphlab.sys_util)

to determine where sys_util.py is located, replace it with the one at the web address above.

Thanks for your patience, everyone. We're working hard to figure it out.

-- Hoyt


User 2809 | 12/13/2015, 5:28:17 AM

Wow - I am grateful to have found this thread. I am also on the Coursera course and spent the last couple of days pulling my hair out trying to fix this on two separate Windows machines. So now I have less than 24 hours to finish the week's assignments. I'm thinking that maybe it would have been a good community service to alert the students to this problem, rather than leave people frustrated about this bug?

The incantations that worked for me (thanks for posting them!) were the ones from earlier in the thread:

<code>import ctypes, inspect, os, graphlab from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', uselasterror=True) kernel32.SetDllDirectoryW.argtypes = (wintypes.LPCWSTR,) srcdir = os.path.split(inspect.getfile(graphlab))[0] kernel32.SetDllDirectoryW(srcdir) </code> As noted, the steps needed seem to be: 1. Shut down any instances of ipython / python / anaconda that have already imported graphlab (started the server) before trying this. In fact, just reboot your PC - this is Windows, after all. 2. Restart a fresh instance of python / ipython notebook 3. Enter the incantations ahead of / in a cell in ipython notebook before / the point that graphlab is imported.

Hope to see this fixed in a release soon. I am frustrated, but appreciate the responsiveness shown in this thread so far! Next time I will start here rather than trying to fix on my own.


User 2772 | 12/13/2015, 12:58:24 PM

@hoytak

I created a new conda env, replaced the file and tried the first fix again, but sadly it did not work.

I still get the runtime error and this in the log for each lambda worker.

Status: Communication Failure. Retry: 3

User 2821 | 12/14/2015, 3:25:23 PM

I got the same error as @Peterrr

My environment:

Windows 7 pro, 12GRAM GraphLab 1.7.1, python2.7.9, Anaconda 2.4.1 (try Anaconda 2.3.0 before)

My error is

import graphlab sa=graphlab.SArray([1,2,3]) sa.apply(lambda x : 2*x)

Such a simple code kept generating lambda worker problem,

I did try: the scripts of @Patrick, upgrade to Python 2.7.11 and downgrade to Python 2.7.9.

But this error is still there, what should I do? Could anyone in Dato team help?

Thanks very much


User 1207 | 12/14/2015, 5:46:41 PM

Hello everyone,

A quick status update from us here at Dato:

We're working very hard to address these issues. The last update fixed all the other known issues we had, but created others. What has been making it difficult for us here to address them is that we haven't been able to reproduce all the failures it seems people are having. We were able to reproduce and fix the DLL issue, which the posted code addresses, but since that did not work for everyone, it seems there are other issues as well.

Our plan is to release a bugfix shortly that incorporates this fix and has a much more informative diagnostic mode that should help us figure out what else is failing.

Thank you all for your patience; we are trying hard to make this better experience.

-- Hoyt


User 2382 | 12/14/2015, 6:28:19 PM

Hi @hoytak & everyone,

[Replacing the sysutil.py corrected my environment.](http://forum.dato.com/discussion/comment/4392/#Comment4392)

In the process I did uninstalled all python and reinstalled the latest (Anaconda2-2.4.1-Windows-x8664.exe)[https://www.continuum.io/downloads] along with GraphLab 1.2.2-2 via pip and then applied the sysutil.py fix.

Now back to productive work...

Thanks, -Myron


User 1207 | 12/14/2015, 7:07:35 PM

Okay, one more thing to try. If some of you having problems could try this and report back, we'd appreciate it. (We have seen that windows and some antivirus programs sometimes flag a program spawning multiple processes simultaneously as suspicious, and then cause problems; this test is to see if this is the case here). This may not work, but if it does, it would help us track down the issue.

Before running anything with graphlab:

import os
os.environ["OMP_NUM_THREADS"] = "1"
import graphlab
graphlab.SArray(range(1000)).apply(lambda x: x)

Thanks! -- Hoyt


User 2772 | 12/14/2015, 7:48:33 PM

@hoytak

I still get the same error, I tried shutting off my antivirus but it still didn't work.

Communication Failure. Retry: 3
1450122297 : ERROR:    (operator():170): Fail launching lambda worker.
1450122297 : ERROR:    (operator():189): Unable to evaluate lambdas. Lambda workers did not start.

I have python installed, but I also have anaconda I will try to uninstall all versions and just keep the one in conda and see if the sys_util.py fixes it like it did for @myron


User 1207 | 12/14/2015, 10:39:09 PM

Hi @Peterrr,

Did you happen to try with the os.environ fix above?

Thanks, -- Hoyt


User 2772 | 12/14/2015, 10:49:29 PM

@hoytak

Yes.


User 2821 | 12/14/2015, 11:32:41 PM

Hi @hoytak, I tried this but it still doesn't work.


User 1207 | 12/15/2015, 12:02:46 AM

@Peterrr @huangphoenix: Thank you.


User 2821 | 12/15/2015, 1:32:11 AM

I tried every method in this thread except downgrade graphlab from 1.7.1 to 1.2.2 as @myron mentioned before.

Could anyone in dato told me if I can downgrade graphlab in command line of anaconda?


User 2382 | 12/15/2015, 2:55:04 AM

All, A correction to my previous (post)[http://forum.dato.com/discussion/comment/4401/#Comment_4401] - GraphLab 1.7.1 using pip was installed. Also, I placed the python path (C:\Anaconda2 for my system) was put first in the system PATH (as opposed to the user PATH variable). Type 'set' at a cmd prompt and verify. Sorry about that, -Myron


User 2821 | 12/15/2015, 10:12:40 AM

Hi for those who are still struggling. I found an intermediate solution for this lambda workers problem.

Here are my code for the classification of the first class ML package

My environment: windows 7 Pro 64 bit, Anaconda 2.4.1 (python 2.7.11), the latest pandas and graphlab_create release

I got stuck when I was taking the home work of the third chapter of the first class of the ML specialization class by UW, and when I did .apply() on SArray or SFrame, it went wrong with the "lambda worker" problem.

So I chose to cheat the system using pandas, here are my code:

import graphlab as gl, pandas as pd products = gl.SFrame('amazon_baby.gl/') products['wordcount'] = gl.textanalytics.countwords(products['review']) pdpandasdata=products.todataframe() pdpandasdata['awesomecount']=pd_pandasdata['wordcount'].apply(awesomecount) %here awesomecount is my own
script to count the word "awesome" products2=gl.SFrame(pd
pandasdata) %I differentiate products2 and products for I might delete products avoiding memory leakage

ok, it looks fine for me now and I can make some REAL work, it took me around two days to temporarily solve this problem.


User 2821 | 12/15/2015, 10:31:21 AM

pitfall of my method is that it needs more memory, especially for pandas dataframe, but in the coursera class, it should be fine. Also, I wish the Dato team can fix this problem ASAP and when we are in capstone in next June, we don't need to worry about it.


User 2821 | 12/15/2015, 10:31:25 AM

pitfall of my method is that it needs more memory, especially for pandas dataframe, but in the coursera class, it should be fine. Also, I wish the Dato team can fix this problem ASAP and when we are in capstone in next June, we don't need to worry about it.


User 1207 | 12/15/2015, 7:24:55 PM

@huangphoenix: We are working hard on a bugfix release that we will be releasing shortly that will hopefully address these issues, or help us to track it down.

You can also do a workaround with the following:

X['column'] = [awesome_count(v) for v in X['column']]

But again, that can cause the memory to significantly blow up.


User 2854 | 12/18/2015, 1:37:30 AM

Hopefully Dato will deploy lambda issue fix before the x-mas break later this month :)


User 2874 | 12/20/2015, 9:59:09 PM

Hello all,

I have another post to add to the ongoing research into this problem. I have followed all of the proposed fixes and none have worked. I'm also in the Coursera class. To add to the potential comlexities, my C drive is a tiny SSD, so nothing python related is there (except for some log files that I can't tell to go other places). Instead, it's on one of my multiple other drives, I:. I am running GraphLab in a separate Anaconda environment, per the pip instructions on the website. Here's my error message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-2404ecb7a089> in <module>()
      1 import graphlab
      2 # Should work
----> 3 graphlab.SArray(range(1000)).apply(lambda x: x)

I:\Program Files (x64)\Anaconda\envs\dato-env\lib\site-packages\graphlab\data_structures\sarray.pyc in apply(self, fn, dtype, skip_undefined, seed)
   1691 
   1692         with cython_context():
-> 1693             return SArray(_proxy=self.__proxy__.transform(fn, dtype, skip_undefined, seed))
   1694 
   1695 

I:\Program Files (x64)\Anaconda\envs\dato-env\lib\site-packages\graphlab\cython\context.pyc in __exit__(self, exc_type, exc_value, traceback)
     47             if not self.show_cython_trace:
     48                 # To hide cython trace, we re-raise from here
---> 49                 raise exc_type(exc_value)
     50             else:
     51                 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Unable to evaluate lambdas. Lambda workers did not start.

I have 3 python executables running before trying to use apply(), and I have 10 running after that. Those 10 stay active the whole time, even after the error message has appeared.


User 1207 | 12/22/2015, 2:25:27 AM

Hey Everyone, thanks for your patience. We have identified 3 potential things that could cause this failure (they were a big pain to repro, believe me), and we're trying to get a fix out shortly for all of you to test. Thanks for being patient with all of these challenges -- Windows development has proven to be a lot trickier than Linux :neutral:


User 1207 | 12/23/2015, 11:46:19 PM

UPDATE (04/11/2016):

The latest version of GLC, 1.8.5, contains fixes for almost all the current known issues. There are two known issues discovered since the release of 1.8.5 that can cause failures, and we plan to fix those ASAP. The first of these deals with possible filename conflicts when running multiple python scripts in parallel that use the same lambda functions, and the second deals with running GLC in virtual containers that do not correctly report the amount of system interprocess shared memory available inside the container.

If you are having issues running GLC 1.8.5, please let us know so that we can figure out the issues and ensure the lambda workers run reliably on every system out there.

Thanks! -- Hoyt


Hello everyone,

Thanks for your patience. Hopefully, we finally have a fix to the issues many of you are having, and we would greatly appreciate your help in testing it and verifying that it works, or, if it does not, in tracking down the remaining issues. I've built a wheel containing fixes to the known issues and posted it at https://s3-us-west-1.amazonaws.com/dato-public-share/GraphLabCreate-1.7.1.94-cp27-none-winamd64.whl.

To install:

If you are using Dato Launcher, start the command prompt from the dato launcher. If not, start your own conda or python environment. Then run

pip install --upgrade https://s3-us-west-1.amazonaws.com/dato-public-share/GraphLab_Create-1.7.1.94-cp27-none-win_amd64.whl

To test:

In your ipython or ipython notebook environment, run:

import graphlab as gl
print gl.SArray(range(1000)).apply(lambda x: x).sum() 

This should print out the sum of these numbers, 499500.

If this works, please leave a comment and let us know! We would love to know that the many hours we spent on this issue have been worth it.

If it fails:

Please run the following code and send us the log files, or, if any of the log files do not get written, then please let us know what happens. All of this will help ensure that we get these issues cleaned up once and for all.

import os
os.environ["GRAPHLAB_LAMBDA_WORKER_DEBUG_MODE"] = "1"
logfile=os.path.expanduser(r"~/graphlab-lambda.log")
os.environ["GRAPHLAB_LAMBDA_WORKER_LOG_FILE"] = logfile
os.environ["OMP_NUM_THREADS"] = "1"

print "Writing logs to ", logfile, " and ", (logfile + "-init")

import graphlab as gl
print gl.SArray(range(1000)).apply(lambda x:x).sum()

This code will produce up to three log files. The first is the server log (printed when the first graphlab command is run) and the two lambda log files printed with the code above. Hopefully these log files will allow us to track down any remaining issues.

Thank you! -- Hoyt & the rest of the Dato team.


User 2854 | 12/24/2015, 12:30:19 AM

Hi hoytak, Installed fixes using pip install. Note: after import graphlab.... [INFO] Using MetricMock instead of real metrics, mode is: QA

I ran through your suggested test and can verify the correct output. Also tested several other applications of apply() lambda and all worked as expected (before fix all generated the runtime exception).

Thanks again for the timely fix!


User 1207 | 12/24/2015, 1:03:34 AM


User 2858 | 12/24/2015, 4:28:47 AM

hey Hoytak, your fix works, :) got 499500 with no errors. all CPU cores jumped up and down and python subprocesses finished correctly. you are a legend cheers Alex


User 2854 | 12/24/2015, 7:14:22 AM

hoytak,
Fix is working as stated earlier, however an observation I'm seeing is that IPython (in the dato-env) will spawn 8 additional python.exe instances as background processes (per Task Manager) after executing the apply() method (w/ or w/o a lambda). Those 8 python process will stick around until the IPython shell is closed.

System/OS info: CPU: Intel Core i7-2600K, 3.4GHz | memory: 8GB | Windows 8.1 Enterprise 64-bit

Code ran in IPython (dato env): import os; import numpy as np; import graphlab as gl sa = gl.SArray([1,2,3]) sa.apply(np.square)

Output: dtype: int Rows: 3 [1L, 4L, 9L]

Any ideas? Thanks.


User 1207 | 12/24/2015, 2:27:47 PM

Hello @CesarO,

Those processes are the lambda workers, so that is to be expected. Because of python's global interpreter lock, this is the only way to get efficient parallelism on .apply(). Once they and the proper communication channels are set up, they are simply idle until the times you need to call apply().

-- Hoyt


User 2854 | 12/24/2015, 6:22:51 PM

@hoytak , thanks for the explanation. Cheers!


User 2902 | 12/25/2015, 5:01:49 PM

Hi @hoytak!

Your latest fix works for me on Windows 8.1. Thanks!

However, the show() method no longer works in IPython Notebook. I set the target to be ipynb but nothing gets displayed (the method executes normally). It works with base graphlab 1.7.1 but not with .94 :-(

And, by the way, Merry Christmas!


User 2874 | 12/26/2015, 5:44:27 PM

Thank you so much for the fix! I am having the same problem as yoctozepto for using show in an ipynb, but that is much preferred over apply not working!


User 2932 | 12/31/2015, 1:18:30 PM

what is the final/resolution for this issue? It is still not working. Tried above mentioned fixes


User 2932 | 12/31/2015, 1:34:13 PM

there were so many solutions back and forth. Please summarize the working solution for the benefit of new person Thanks in advance


User 2854 | 12/31/2015, 6:45:26 PM

@rajendra, did you look at top of first page where it reads "Best Answer"? There is a fix (install GL version 1.7.1.94) for the lambda bug + additional instructions (however note that it looks like this ver will break some functionality in show() method).


User 2932 | 1/1/2016, 5:03:55 AM

Hi @CesarO Thanks a lot. This worked. It was my mistake, that I did not notice it. once again thanks. @All Wish you all a very happy new year 2016.


User 2941 | 1/2/2016, 6:27:53 AM

I also have same issue. Hence attaching the log file as asked. Other details: O/S - Wind 7 64 bit Using Dato Launcher After using SFrame - keep getting * on Left hand side bar for each of the commands executed next.![] CODE TESTED- import ctypes, inspect, os, graphlab from ctypes import wintypes kernel32 = ctypes.WinDLL('kernel32', uselasterror=True) kernel32.SetDllDirectoryW.argtypes = (wintypes.LPCWSTR,) srcdir = os.path.split(inspect.getfile(graphlab))[0] kernel32.SetDllDirectoryW(srcdir)

graphlab.SArray(range(1000)).apply(lambda x: x)

Please help.


User 2941 | 1/2/2016, 6:42:11 AM

Good fellas, I also had an automatic windows update some 2-3 weeks back and have issues since than (though not 100% sure it is because of Windows update). Everything was working for a month.. before the update!


User 2941 | 1/2/2016, 8:40:52 PM

After couple of installations it seems the problem is in starting of Unity Server (unity_server) while launching SFrame or SArray. Will wait for any response.


User 1207 | 1/3/2016, 10:15:18 PM

@jain -- Did you try running the egg in the pinned answer at the beginning of this forum thread? However, if you are having issues starting unity_server, that is likely a different issue, but still one we want to resolve. In that case, would you mind posting it to a new forum thread in case other people are having the same issue?

Thanks! -- Hoyt


User 2902 | 1/4/2016, 4:56:50 AM

@hoytak -- What about the .show() issue? In .94 it stopped working in IPython Notebook...


User 2772 | 1/4/2016, 12:10:58 PM

Just wanted to say that I installed Ubuntu 14.04 and set up the anaconda dato-env there and it worked without any issues.


User 2941 | 1/4/2016, 2:24:38 PM

Hi Hoyt, Will create a separate thread for it. Many thanks!


User 1207 | 1/6/2016, 10:58:21 PM

@yoctozepto, we'll be releasing an updated version shortly that will include a fix to that issue, as well as the pylambda worker fixes.


User 2902 | 1/7/2016, 6:34:31 AM

@hoytak, thanks for your response! That is some great news! I am looking forward to that release.


User 2941 | 1/8/2016, 4:34:32 PM

@hoytak, We are quite excited about the next release.. and waiting for good news!


User 2993 | 1/11/2016, 12:30:22 AM

The fix worked:)


User 3116 | 1/24/2016, 5:30:36 AM

Another Coursera student here, I'm having the same problem. I applied the fix posted under "Best Answer" and still receive the same results. I attached the log files requested to this post (had to change the file name of *.log-init to get it to upload)

Thanks for any help you can provide!


User 2941 | 1/24/2016, 9:58:54 AM

Hi Hoytak, Can you please confirm about the next release.. released or not. Hope this issue to be fixed.. :wink:

Best regards, Jain


User 2854 | 1/24/2016, 10:05:09 AM

Interesting b/c the original fix (in GL ver 1.7.1.94) worked to resolve the lambda bug for me. And it is also working (in addition to fixing regression in show() function issue) when I upgraded to GL ver 1.8.


User 2941 | 1/25/2016, 2:01:09 AM

Thanks CesarO.


User 2983 | 2/23/2016, 5:58:22 PM

Hello everyone, I hope the thread is still active. I have been doing the Coursera Course on Machine Learning and as a part of it trying to run the apply() method in Graphlab. However getting the all famous Lambda workers did not start I bumped into this post and tried the solutions given in the threads. I did install the *.whl file given in this thread, post which I ran the following import os os.environ["OMPNUMTHREADS"] = "1" import graphlab graphlab.SArray(range(1000)).apply(lambda x: x)

and was able to successfully run the same. However, whenever I try to do the same with my SFrames I again get the Lambda workers did not start error.


User 1207 | 2/24/2016, 11:33:40 PM

@codeahead14 -- could you tell us a bit more about your system? How many cores do you have? Also, which version of SFrame or Graphlab create are you using? Also, could you send or attach your server log files? The path to these is printed at the top when the server starts. It's likely a new issue...

Thanks! -- Hoyt


User 3531 | 3/14/2016, 2:46:59 PM

Hi hoytak,

I meet the same issue in Ubuntu14.04 with Anaconda2.5.0 and Python2.7.11.

RuntimeError Traceback (most recent call last) <ipython-input-3-e7cb76710529> in <module>() 5 return country 6 ----> 7 sf['Country'].apply(transform_country)

/mirror/anaconda2/envs/dato-env/lib/python2.7/site-packages/graphlab/datastructures/sarray.pyc in apply(self, fn, dtype, skipundefined, seed) 1699 1700 with cythoncontext(): -> 1701 return SArray(proxy=self.proxy.transform(fn, dtype, skip_undefined, seed)) 1702 1703

/mirror/anaconda2/envs/dato-env/lib/python2.7/site-packages/graphlab/cython/context.pyc in exit(self, exctype, excvalue, traceback) 47 if not self.showcythontrace: 48 # To hide cython trace, we re-raise from here ---> 49 raise exctype(excvalue) 50 else: 51 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. LOG_FATAL encountered

And here is the log. What should I do? Thanks!


User 1207 | 3/14/2016, 5:48:39 PM

Hello @qyy0180,

Thanks for the detailed message and log. Something seems to be wrong with your /var/tmp/ directory. Could you tell me what ls -l /var/tmp says? It may be a permissions thing. The other possibility is that one of your disks is full, in which case running df will tell you if a disk is full (marked as 100% full). However, I suspect it is a permissions issue.

One way around this is to set the environment variable GRAPHLAB_CACHE_FILE_LOCATIONS to a different directory. This can be done by running gl.set_runtime_config("GRAPHLAB_CACHE_FILE_LOCATIONS", "<path to temp dir>") when you first start graphlab, or by setting it as an environment variable before you start python. For example, I ran gl.set_runtime_config("GRAPHLAB_CACHE_FILE_LOCATIONS", "/home/hoytak/tmp") to test this.

Thanks! -- Hoyt


User 3531 | 3/15/2016, 1:09:23 AM

Hi @hoytak ,

The message ls -l /var/tmp says drwxrwxr-x 4 1001 1001 4096 . It works after I set the environment variable GRAPHLABCACHEFILE_LOCATIONS to a different directory. So I should run this command whenever I start graphlab?

Thanks!


User 1207 | 3/15/2016, 7:52:39 PM

Hey @qyy0180,

It appears, for some reason, that the ownership of the /var/tmp directory got messed up. It should say your user name in there instead of 1001. Changing this (using sudo chown -R <username> /var/tmp/) should fix things).


User 3531 | 3/16/2016, 2:04:29 AM

Hi @hoytak,

Thanks! It works well now.


User 3703 | 3/17/2016, 6:55:47 PM

Just got my GraphLab Create running for the Coursera class and hit the "RuntimeError: Runtime Exception. Cannot evaluate lambda. Lambda workers cannot not start." error when using .apply(). I'm using GLC 1.8.4 on a Mac OSX Yosemite 10.10.5. I went through the posts, but must've lost what the ultimate fix was. Thanks.


User 1207 | 4/11/2016, 9:23:24 PM

Hello @tjsanfo,

Sorry to not get back to you -- I missed your comment earlier. Does 1.8.5 fix any of the issues for you? There were several other bugs reported with fixes that went in to 1.8.5. If not, then let's figure out what the issues are.

Thanks! -- Hoyt


User 523 | 4/13/2016, 5:06:52 AM

I have the same problem in the same course:

I ran this code or word in words:

products[word] = products['wordcount'].apply(lambda count: selectedword_count(count, word))

Got this ERROR:

"RuntimeError: Runtime Exception. Cannot evaluate lambda. Lambda workers cannot not start."

I use the launcher with GraphLab-Create (1.8.5)

on A Mac

del Name: MacBook Pro Model Identifier: MacBookPro7,1 Processor Name: Intel Core 2 Duo Processor Speed: 2.4 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache: 3 MB Memory: 8 GB Bus Speed: 1.07 GHz Boot ROM Version: MBP71.0039.B0E SMC Version (system): 1.62f7 Serial Number (system): Hardware UUID: CB-1 Sudden Motion Sensor: State: Enabled


User 4823 | 4/15/2016, 5:23:39 PM

Appears I have the same problem.

Attempted to run the following - tmp = graphlab.SArray([1., 2., 3.]) tmp_cubed = tmp.apply(lambda x: x**3) print tmp print tmp_cubed

And got this error - `


RuntimeError Traceback (most recent call last) <ipython-input-4-121df277dd60> in <module>() 1 tmp = graphlab.SArray([1., 2., 3.]) 2 tmpcubed = tmp.apply(lambda x: x**3) 3 print tmp 4 print tmpcubed

/Users/jonathanmoore/anaconda/envs/dato-env/lib/python2.7/site-packages/graphlab/datastructures/sarray.pyc in apply(self, fn, dtype, skipundefined, seed) 1699 1700 with cythoncontext(): -> 1701 return SArray(proxy=self.proxy.transform(fn, dtype, skip_undefined, seed)) 1702 1703

/Users/jonathanmoore/anaconda/envs/dato-env/lib/python2.7/site-packages/graphlab/cython/context.pyc in exit(self, exctype, excvalue, traceback) 47 if not self.showcythontrace: 48 # To hide cython trace, we re-raise from here ---> 49 raise exctype(excvalue) 50 else: 51 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Cannot evaluate lambda. Lambda workers cannot not start. `

MAC OS X 10.11.2 Just upgraded to GraphLab Create - v1.8.5 from v1.8.2 . I did not experience this error with the previous version.

Interestingly - I tried running the test found earlier in this thread in a new notebook and it seemed to work fine.

running - import graphlab as gl print gl.SArray(range(1000)).apply(lambda x: x).sum() resulted in - 499500


User 4 | 4/15/2016, 9:37:54 PM

Hi @JonMoore, can you please post your OS and GraphLab Create logs if available (or submit them to our support ticketing system)? This will help diagnose the specific issue here. The GraphLab Create log file location is printed to standard output when GraphLab Create first starts within the Python process.


User 4922 | 4/17/2016, 12:31:02 AM

I have a similar problem with GLC. Kindly assist. When I try to run the code below on my Mac: fs['Country'].apply(transform_country) I get this error below: RuntimeError Traceback (most recent call last) <ipython-input-34-cb77c638b4b7> in <module>() 1 import graphlab as gl ----> 2 print gl.SArray(range(1000)).apply(lambda x: x).sum()

/Users/<<username>>/anaconda/envs/dato-env/lib/python2.7/site-packages/graphlab/datastructures/sarray.pyc in apply(self, fn, dtype, skipundefined, seed) 1699 1700 with cythoncontext(): -> 1701 return SArray(proxy=self.proxy.transform(fn, dtype, skip_undefined, seed)) 1702 1703

/Users/<<username>>/anaconda/envs/dato-env/lib/python2.7/site-packages/graphlab/cython/context.pyc in exit(self, exctype, excvalue, traceback) 47 if not self.showcythontrace: 48 # To hide cython trace, we re-raise from here ---> 49 raise exctype(excvalue) 50 else: 51 # To show the full trace, we do nothing and let exception propagate

RuntimeError: Runtime Exception. Cannot evaluate lambda. Lambda workers cannot not start.


User 4823 | 4/17/2016, 5:15:01 PM

@Zach. I'm happy to provide information but I'm afraid I might need a bit of guidance to provide what you have requested. I'm guessing the following shows the GraphLab Create log location?

2016-04-16 13:10:04,603 [INFO] graphlab.cython.cyserver, 176: GraphLab Create v1.8.5 started. Logging: /tmp/graphlabserver_1460826602.log

Also, I'm using a MAC and there are a ton of log files. Any more specific guidance you can give me on what you are looking for?

I tried running the code again and got the same error so it seems I can repeat it and recover the logs for you if I'm sure what files to send.


User 4 | 4/18/2016, 2:07:07 AM

Hi @JonMoore and @Segun (and anyone else who is still having issues starting lambda workers):

Another Dato engineer has provided a script that will help diagnose these issues. Please run the script located here with your GraphLab Create installation. This script will produce a zip file of output and print out the location of that file. Please open a support ticket and after you get an e-mail from our ticketing system, reply to the e-mail and attach that zip file, and we'll be able to better diagnose each of the issues that are happening here. Thanks very much!


User 4823 | 4/18/2016, 3:02:05 PM

Got it. I am forwarding output now.

Just a heads up to MAC users - the output zip file is a hidden file and you will need to view hidden files in order to find it.


User 4922 | 4/22/2016, 2:15:42 PM

Hi @Zack, I ran the script and successfully generated the ZIP file. Unfortunately the support ticket never dropped in my mail. Do we have a solution for Mac users? I can't continue the interesting courses using my local machine. Kindly provide updates. Thanks


User 4 | 4/22/2016, 9:59:50 PM

Hi @Segun, we had a temporary issue with our support ticket system where e-mails were not going through. Can you please try submitting the ticket again? Thanks!


User 4922 | 4/25/2016, 9:36:29 AM

Hello Zack, thanks for the response. I have sent the ZIP report. Regards


User 9 | 4/26/2016, 8:29:02 PM

I created a post that describes how to send Dato a lamba test report here.