Unable to get Sframe's .apply and .filter functions working

User 2036 | 6/8/2015, 3:58:16 PM

Hi All

Most functions of Graph Lab's methods work great for me except functions that depend on lambda operator.

For example, sf['age_double'] = sf['age'].apply(lambda age: age*2)

Just hangs the computer and nothing happens even after 30 minutes. I don't know what is wrong. My guess is that it has something to do with environment. I am using Macbook pro for my analysis and followed the Graphlab instructions exactly.

I use basic python and anaconda.

Can some one suggest a robust pipeline that would enable me to troubleshoot this problem?

Regards Revanth

Comments

User 19 | 6/8/2015, 4:09:23 PM

Hi Revanth,

Is this also true for small synthetic examples, such as sf = gl.SFrame({'age': [1, 2, 3]})?

Also, to facilitate debugging this issue, can you send the last 30 or so lines of the logfile? The log location should be mentioned as soon as you do your first command involving GraphLab Create.

Thanks! Chris


User 2036 | 6/9/2015, 1:50:50 AM

Hi Chris, Thanks for the response. I am getting the same problem even with small Sframes like the one you specified. The lambda workers don't seem to be doing much when ran "top" command on the command line to see which process is actively executing.

I am not sure how to access the log file with the following message. [INFO] Start server at: ipc:///tmp/graphlabserver-706 - Server binary: /usr/local/lib/python2.7/site-packages/graphlab/unityserver - Server log: /tmp/graphlabserver1433610159.log

I tried to locate the log file without much success. I do not know how to get to the log file with the above information.


User 19 | 6/9/2015, 4:21:25 AM

Hi Revanth,

Are there any files in /tmp pertaining to graphlab_server? Sometimes the actual filename is slightly different, e.g., there's a log.0 suffix.

Chris


User 2036 | 6/9/2015, 1:59:10 PM

Hi Chris! Thanks for you help. I figured out how to find the log. Here's exactly what I ran:

import graphlab as gl sf = gl.SFrame({'age': [1, 2, 3]}) sf['age_double'] = sf['age'].apply(lambda x: 2*x)

I ran this with Python 2.7.9, GraphLab-Create 1.4.0. To be clear, this is not an Anaconda install; is Anaconda necessary? When I run the above it hangs and I have to use Ctrl-Z to get out and then kill the Python process. I can successfully create the sf, but the apply part causes the problem. The tail of the log is below. Any ideas? Thanks!

1433857461 : INFO: (register_toolkit_function:41): Function entry 1433857461 : INFO: (load_toolkit:489): Adding class : demo_class 1433857461 : INFO: (load_toolkit:489): Adding class : demo_vector 1433857461 : INFO: (register_toolkit_class:26): Function entry 1433857461 : INFO: (register_toolkit_class:26): Function entry 1433857461 : INFO: (start:215): Function entry 1433857461 : INFO: (create_arrays_for_writing:228): Opening Frame for writing to with 1 segments and 0 columns 1433857461 : INFO: (construct_from_vector:78): Function entry 1433857461 : INFO: (do_copy:373): Function entry 1433857461 : INFO: (transform:390): Function entry 1433857461 : INFO: (spawn_worker:50): Start lambda worker at ipc:///var/tmp/graphlab-revanthgarlapati/2710/000000 using binary: /usr/local/lib/python2.7/site-packages/graphlab/pylambda_worker 1433857461 : INFO: (spawn_worker:50): Start lambda worker at ipc:///var/tmp/graphlab-revanthgarlapati/2710/000001 using binary: /usr/local/lib/python2.7/site-packages/graphlab/pylambda_worker 1433857461 : INFO: (spawn_worker:50): Start lambda worker at ipc:///var/tmp/graphlab-revanthgarlapati/2710/000002 using binary: /usr/local/lib/python2.7/site-packages/graphlab/pylambda_worker 1433857461 : INFO: (spawn_worker:50): Start lambda worker at ipc:///var/tmp/graphlab-revanthgarlapati/2710/000003 using binary: /usr/local/lib/python2.7/site-packages/graphlab/pylambda_worker 1433857461 : INFO: (launch:116): Launched process with pid: 2715 1433857461 : INFO: (launch:116): Launched process with pid: 2716 1433857461 : INFO: (launch:116): Launched process with pid: 2717 1433857461 : INFO: (launch:116): Launched process with pid: 2718


User 19 | 6/9/2015, 4:32:51 PM

Can you say a bit more about your installation process? Do you have multiple versions of python on your computer? Have you tried installing into a virtualenv or after installing anaconda/miniconda?


User 2036 | 6/9/2015, 8:24:12 PM

I followed exactly these directions:

virtualenv dato-env source dato-env/bin/activate pip install --upgrade pip pip install --upgrade --no-cache-dir https://get.dato.com/GraphLab-Create/1.4.0/...

I do also have a system Python 2.7.2 in /usr/bin/python, but I run 2.7.9 from /usr/local/bin/python.

I'll try it with Anaconda and see if that works better.


User 19 | 6/9/2015, 8:55:16 PM

Ah, I see. In the past we've seen this issue when users have multiple Python versions installed, so please do try Anaconda/miniconda and let me know if that works better!


User 2036 | 6/9/2015, 9:15:34 PM

Hi Chris I tried the following Anaconda installation instructions but no improvement in the results.

conda create -n dato-env python=2.7 source activate dato-env conda update pip pip install --upgrade --no-cache-dir https://get.dato.com/GraphLab- Create/1.4.0/a@b.com/ ####-####-####-####-####-####-####- ####/GraphLab-Create-License.tar.gz

Do you suggest that I make sure there is only one version of python on the Macbook? I have 2 versions of Python and Anaconda installed right now.

OR could it be something else?


User 19 | 6/9/2015, 10:32:30 PM

Can you report your PYTHONPATH? We want to make sure that it's actually using your conda install.

Thanks!


User 2036 | 6/10/2015, 1:33:10 PM

Hi Chris, The following is my PYTHONPATH that I get using the print sys.path command: ['', '/Users/revanthgarlapati/anaconda/lib/python27.zip', '/Users/revanthgarlapati/anaconda/lib/python2.7', '/Users/revanthgarlapati/anaconda/lib/python2.7/plat-darwin', '/Users/revanthgarlapati/anaconda/lib/python2.7/plat-mac', '/Users/revanthgarlapati/anaconda/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/revanthgarlapati/anaconda/lib/python2.7/lib-tk', '/Users/revanthgarlapati/anaconda/lib/python2.7/lib-old', '/Users/revanthgarlapati/anaconda/lib/python2.7/lib-dynload', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages/Sphinx-1.2.3-py2.7.egg', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages/aeosa', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages/cryptography-0.8-py2.7-macosx-10.5-x86_64.egg', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages/setuptools-14.3-py2.7.egg', '/Users/revanthgarlapati/anaconda/lib/python2.7/site-packages/IPython/extensions']

It appears that conda installation is being used.

Is there any other check I need to make?