SFrame apply fails with lambda functions, IPython and imported function

User 5220 | 5/23/2016, 3:20:06 PM

Hi, I'm using GraphLab create 1.9 with a Windows 8.1 Pro 64 bit computer, anaconda 4.0.6 and Python 2.7.11. Using IPython 4.1.2 from command line, if I run main.py with the following code inside:

# main.py file
import graphlab as gl
import matplotlib.pyplot as plt

from datetime import datetime
from dateutil import tz
from candles_library import * # it's my personal library

qq = gl.SFrame("SP500_5min_new.csv")
qq['datetime'] = qq['utctimestamp'].apply(lambda x: from_utc_to_local(x, tz.gettz('America/New_York'))) # run-time error here!

after the executing of last instruction I got this error:

**RuntimeError: Runtime Exception. Runtime Exception. Exception in python callback function evaluation: ImportError('No module named candleslibrary',): Traceback (most recent call last): File "graphlab\cython\cypylambdaworkers.pyx", line 393, in graphlab.cython.cypylambdaworkers.initlambda File "graphlab\cython\cypylambdaworkers.pyx", line 133, in graphlab.cython.cypylambdaworkers.lambdaevaluator.init ImportError: No module named candles_library**

Function fromutctolocal is defined inside candleslibrary as follow::

# candles_library.py file
def from_utc_to_local(utc_ts, local_tz):
	utc_tz = tz.gettz('UTC')
	ts = datetime.utcfromtimestamp(utc_ts)
	ts = ts.replace(tzinfo=utc_tz)
	ts = ts.astimezone(local_tz)
	return ts

Conversely, if I run the program outside IPython, from command line as:

python main.py

the code run smoothly with no error at all.

It seems that SFrame.apply method called inside IPython, does not load functions declared in imported modules, the strange thing is that such an issue, is tied to IPython only.

Comments

User 1774 | 5/23/2016, 6:29:37 PM

Hi, This is a known limitation of the pickle mechanism (as your function is pickled when it is passed to SFrame.apply). To solve it quickly, you can simply update your PYTHONPATH environment variable to include the directory from which you are running this code (you can do it from the cmd just before running the python/ipython shell).


User 1207 | 5/23/2016, 9:21:50 PM

Hello @nalon99,

It seems that ipython is not properly propagating the current working directory on the system path to our server. Does it work if you start your code with:

import os, sys sys.path.append(os.getcwd()) import graphlab as gl

If that works, then we'll have a fix out shortly.

Thanks! -- Hoyt


User 5220 | 5/25/2016, 1:52:50 PM

Hi, I've tried the same test as described in my first post, with the same computer, same environment and now it works both from command line and IPython. I did not even set PYTHONPATH or sys.path.append(os.getcwd). Today apply method works properly without rising runtime error. I don't know why it didn't work 2 days ago, but today it works. Anyway, thanks for your answers.


User 5220 | 5/25/2016, 3:35:13 PM

Well, I've rebooted Windows and now, runtime error has arisen again...

So, adding the following lines as @toyak suggested, do solve that issue!

import os, sys
sys.path.append(os.getcwd())
import graphlab as gl