Current Steps for Lambda Worker Errors

User 1207 | 6/3/2016, 7:08:40 PM

Getting the lambda workers to work reliably in all sorts of environments is a huge challenge, and while we have been working steadily to resolve all the issues encountered, there are numerous possible ways they can fail. If you are encountering issues getting the lambda workers to start, or if they start intermittently, then please go through the following steps. ​ 1. Install or upgrade to Graphlab Create 1.10. There is some new features in there make things more robust, as well as new features we've put in there to help diagnose and work around issues encountered. You can see your current version by typing import graphlab as gl ; gl.version in the python/ipython terminal. If upgrading to version 1.10 works, then you're all set!

2. If not, run the following code right when you start python or ipython notebook:

import os
os.environ["GRAPHLAB_DISABLE_LAMBDA_SHM"] = "1"
os.environ["GRAPHLAB_FORCE_IPC_TO_TCP_FALLBACK"] = "1"
import graphlab as gl

3. Test out your lambda worker code in this environment. If it works, then you can make the above configuration permanent by running:

gl.sys_util.write_config_file_value("GRAPHLAB_DISABLE_LAMBDA_SHM", "1")
gl.sys_util.write_config_file_value("GRAPHLAB_FORCE_IPC_TO_TCP_FALLBACK", "1")

Note that this can be undone by setting these to "0" instead of "1", or by editing the file given by gl.sys_util.get_config_file().

4. If the lambda workers do not work after trying step 1, then there are two things we would very much appreciate you do to help us track down the issue.

4.1. First, execute the following code in a clean python shell, where you have not yet imported graphlab create. At the end of this code, it prints out the path to a zip file that, if you could send it to us, will help us diagnose the issue. Please create a support ticket and submit the zip file at https://dato.com/support/create-support-ticket.html.

import os
os.environ["GRAPHLAB_LAMBDA_WORKER_DEBUG_MODE"] = "1"
os.environ["GRAPHLAB_DISABLE_LAMBDA_SHM"] = "1"
os.environ["GRAPHLAB_FORCE_IPC_TO_TCP_FALLBACK"] = "1"
import graphlab as gl
os.environ["GRAPHLAB_LAMBDA_WORKER_LOG_FILE"] = gl.get_server_log_location() + "_lambda"
gl.sys_util.test_pylambda_worker()

4.2. Second, after running the code in step 4.1, run your lambda worker code in the same python shell. If it fails, then please submit all the log files to us through our support ticket site at https://dato.com/support/create-support-ticket.html. You can get these files by running

import glob
print("Please send the following files to Dato: %s" 
          % (", ".join(glob.glob(gl.get_server_log_location() + "*"))))

​​Please let me know if this helps! ​ ​Thanks, ​-- Hoyt

No Comments