Executing Jobs in EC2 Failed

User 1408 | 3/8/2015, 1:19:58 AM

Hi,

I tried to follow the tutorial "Executing Jobs in EC2", but it failed for some reason. It seems there was some bug in the old version of Werkzeug. Attached is the execution log from the ec2 instance:

[INFO] Commander running. [INFO] Checking environment variables. [INFO] Running on http://0.0.0.0:9004/ (Press CTRL+C to quit) [INFO] Restarting with stat [INFO] Commander running. [INFO] Checking environment variables. [INFO] /submit called [INFO] Required packages: [] [INFO] Start server at: ipc:///tmp/graphlabserver-1420 - Server binary: /usr/local/lib/python2.7/dist-packages/graphlab/unityserver - Server log: /tmp/graphlabserver1425775777.log [INFO] GraphLab Server Version: 1.3.0 [INFO] Job file saved: /tmp/tmpRFLmLU [INFO] Deserializing job: /tmp/tmpRFLmLU [INFO] 128.208.7.9 - - [08/Mar/2015 00:49:40] "POST /submit HTTP/1.1" 200 - Traceback (most recent call last): File "/tools/init/jobsdaemons/commander.py", line 368, in <module> app.run(host='0.0.0.0', port=COMMANDERPORTNUM, debug=True, threaded=True) File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 772, in run runsimple(host, port, self, **options) File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 622, in runsimple reloadertype) File "/usr/local/lib/python2.7/dist-packages/werkzeug/reloader.py", line 269, in runwithreloader reloader.run() File "/usr/local/lib/python2.7/dist-packages/werkzeug/reloader.py", line 159, in run for filename in chain(itermodulefiles(), self.extrafiles): File "/usr/local/lib/python2.7/dist-packages/werkzeug/reloader.py", line 70, in itermodulefiles for packagepath in getattr(module, 'path', ()): File "/usr/lib/python2.7/dist-packages/six.py", line 116, in getattr module = self.resolve() File "/usr/lib/python2.7/dist-packages/six.py", line 105, in resolve return importmodule(self.mod) File "/usr/lib/python2.7/dist-packages/six.py", line 76, in importmodule import(name) ImportError: No module named winreg [INFO] Writting state (Running) to cse544-123, logs/add-Mar-07-2015-16-48-08-1425775688.7//status Exception in thread Thread-5: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrapinner self.run() File "/tools/init/jobsdaemons/commander.py", line 180, in run rttask.serialize(taskpath) File "/usr/local/lib/python2.7/dist-packages/graphlab/deploy/task.py", line 762, in serialize pickler.dump(self) File "/usr/local/lib/python2.7/dist-packages/graphlab/glpickle.py", line 293, in dump cloudpickle.CloudPickler.dump(self, obj) File "/usr/local/lib/python2.7/dist-packages/graphlabutil/cloudpickle.py", line 154, in dump return pickle.Pickler.dump(self, obj) File "/usr/lib/python2.7/pickle.py", line 224, in dump self.save(obj) File "/usr/lib/python2.7/pickle.py", line 331, in save self.savereduce(obj=obj, *rv) File "/usr/local/lib/python2.7/dist-packages/graphlabutil/cloudpickle.py", line 645, in savereduce save(state) File "/usr/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python2.7/dist-packages/graphlabutil/cloudpickle.py", line 195, in savedict pickle.Pickler.savedict(self, obj) File "/usr/lib/python2.7/pickle.py", line 649, in savedict self.batchsetitems(obj.iteritems()) File "/usr/lib/python2.7/pickle.py", line 681, in batchsetitems save(v) File "/usr/lib/python2.7/pickle.py", line 331, in save self.savereduce(obj=obj, *rv) File "/usr/local/lib/python2.7/dist-packages/graphlabutil/cloudpickle.py", line 645, in savereduce save(state) File "/usr/lib/python2.7/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/lib/python2.7/dist-packages/graphlabutil/cloudpickle.py", line 195, in savedict pickle.Pickler.save_dict(self, obj) File "/usr/lib/pythHtmlx�I�M! ��7# ++����FYI: If you are using Anaconda and having problems with NumPyHello everyone,

I ran into an issue a few days ago and found out something that may be affecting many GraphLab users who use it with Anaconda on Windows. NumPy was unable to load, and consequently everything that requires it (Matplotlib etc).

It turns out that the current NumPy build (1.10.4) for Windows is problematic (more info here).

Possible workarounds are downgrading to build 1.10.1 or forcing an upgrade to 1.11.0 if your dependencies allow. Downgrading was easy for me using conda install numpy=1.10.1

Thanks for your attention!

RafaelMarkdown558,824,8414L���4L���179.110.206.156179.110.206.1564P�}��Xj�8\j�1str�"��\j�Xj��\j�8bj�րi�1(׀i��g��b�j����Xj�\j�Xj�8\j�1.hpp(decrementdistributedcounter:787): Distributed Aggregation of likelihood. 0 remaining. INFO: distributedaggregator.hpp(decrementdistributedcounter:793): Aggregate completion of likelihood Likelihood: -3.22336e+08 INFO: distributedaggregator.3HLABDISABLELAMBDA_SHM"] = "1" os.environ["GRAPHLABFORCEIPCTOTCP_FALLBACK"] = "1" import graphlab as gl

3. Test out your lambda worker code in this environment. If it works, then you can make the above configuration permanent by running:

gl.sys_util.write_config_file_value("GRAPHLAB_DISABLE_LAMBDA_SHM", "1")
gl.sys_util.write_config_file_value("GRAPHLAB_FORCE_IPC_TO_TCP_FALLBACK", "1")

Note that this can be undone by setting these to "0" instead of "1", or by editing the file given by gl.sys_util.get_config_file().

4. If the lambda workers do not work after trying step 1, then there are two things we would very much appreciate you do to help us track down the issue.

4.1. First, execute the following code in a clean python shell, where you have not yet imported graphlab create. At the end of this code, it prints out the path to a zip file that, if you could send it to us, will help us diagnose the issue. Please create a support tick^j�8bj�2�"��bj�^j��bj�8�j�րi�2(׀i����^j�8bj�2

Comments

User 1178 | 3/8/2015, 8:31:48 PM

Hi Edward,

There is a known issue with the cloud pickler we use. The short story is that 'six' puts moved modules into global sys.modules list that user never imported explicitly.

For now, if you do not import any python package that depends on "six", you should be fine. We have a fix on the way and will be in next GraphLab Create version update.

Thanks!

Ping


User 1408 | 3/8/2015, 10:10:55 PM

Cool. I was just running the add tutorial that I dont believe imported any python package.


User 1178 | 3/10/2015, 3:04:21 AM

Hi Edward,

There was a recent regression in running job in EC2 and the issue has now been fixed. Please try follow the tutorial again.

Thanks!

Ping