SFrame creation failing in celery worker; hangs no error shown

User 481 | 6/2/2015, 9:42:28 PM

Hi, I can create an SFrame in the python environment however when I run the same code inside of a celery worker function the SFrame creation fails; it hangs and does not even give an error message. Has anyone encountered this before?

Manas

Comments

User 1394 | 6/2/2015, 10:22:51 PM

Hey Manas -

Can you share a code snippet so we can reproduce on our end? One important thing to help debug would be to confirm what Python environment the Celery worker is using when invoking GraphLab Create APIs.

Thanks,

Rajat


User 481 | 6/2/2015, 11:47:07 PM

So I will try to recreate the issue

  • task.py from app import myapp @myapp.task(name='myapp.receiver') def receivemsg(msg): m = json.loads(msg) get_data_for_tenant(m['tenant']) def get_data_for_tenant(tenant): votes = get_votes(tenant) # get a list of vote objects # compose SFrame from votes import graphlab as gl user_ids = map(lambda vote: vote.prop('creator').eid.encode('ascii', 'ignore'), votes) idea_ids = map(lambda vote: vote.prop('object').eid.encode('ascii', 'ignore'), votes) ratings = map(lambda vote: vote.prop('valuation'), votes) sf = SFrame({'user_id':user_ids, 'idea_id':idea_ids, 'rating':ratings}) return sf

  • conf.py BROKER_URL = 'amqp://' CELERY_IMPORTS = ('_tasks') CELERYBEAT_SCHEDULER='djcelery.schedulers.DatabaseScheduler'

  • app.py from celery import Celery myapp = Celery('appname') myapp.config_from_object(conf) myapp.worker_main('-B', '-S', 'djcelery.scheduler.DatabaseScheduler') Run worker $python app.py

The getdatafor_tenant function works perfectly in python env but not when running in a celery worker.


User 1394 | 6/3/2015, 1:50:59 AM

Thanks for sharing this, we will try to repro on our side. However, can you share any of the Python that the Celery worker is using? Things like the PYTHONPATH from inside the Celery worker. My hunch is that the Celery worker is using a different site-packages/PYTHONPATH when executing.


User 481 | 6/3/2015, 7:44:28 PM

Hi Rajat,

I checked the python version and python path from inside the celery worker by making it print first. It is the same that version and python path.


User 481 | 6/5/2015, 10:30:12 PM

After some diggin around, it seems like import of graphlab package itself is failing. If I change the getdatafor_tenant(tenant) to the following

def get_data_for_tenant(tenant): print "Importing graphlab ..." import graphlab as gl print "done" The celery worker hangs after printing "Importing graphlab". So it seems it has got something to do with the copy of graphlab in memory.


User 19 | 6/8/2015, 7:10:41 PM

Hi Manas,

Do you have access to any logs from this worker?

Thanks, Chris


User 481 | 6/8/2015, 8:00:08 PM

This is how I am testing graphlab import from within celery worker; log is set to DEBUG - celeryapps.py from celery import Celery testapp = Celery('examples') if __name__ == '__main__': args = ['--detach', '--loglevel=DEBUG', '-B'] testapp.worker_main(argv=args) - celerytasks.py from celeryapps import testapp @testapp.task(name='import_graphlab') def import_graphlab(msg): print "message received: ", msg import graphlab as gl print "... done" print gl - run.py from celeryapps import testapp from celerytasks import import_graphlab import json import_graphlab.delay(json.dumps('hellow'))

Start the worker $python celeryapps.py Log =

! ... ! [2015-06-08 20:09:28,253: DEBUG/MainProcess] | Worker: Starting Hub ! [2015-06-08 20:09:28,253: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,253: DEBUG/MainProcess] | Worker: Starting Pool ! [2015-06-08 20:09:28,276: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,276: DEBUG/MainProcess] | Worker: Starting Beat ! [2015-06-08 20:09:28,278: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,281: DEBUG/MainProcess] | Worker: Starting Consumer ! [2015-06-08 20:09:28,281: DEBUG/MainProcess] | Consumer: Starting Connection ! [2015-06-08 20:09:28,287: INFO/Beat] beat: Starting... ! [2015-06-08 20:09:28,297: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchangeexchangebindings': True, u'connection.blocked': True, u'authenticationfailureclose': True, u'basic.nack': True, u'perconsumerqos': True, u'consumerpriorities': True, u'consumercancelnotify': True, u'publisherconfirms': True}, u'cluster_name': u'rabbit@administrators-MacBook-Air-4', u'platform': u'Erlang/OTP', u'version': u'3.4.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'enUS'] ! [2015-06-08 20:09:28,299: DEBUG/MainProcess] Open OK! ! [2015-06-08 20:09:28,300: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// ! [2015-06-08 20:09:28,300: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,300: DEBUG/MainProcess] | Consumer: Starting Events ! [2015-06-08 20:09:28,306: DEBUG/Beat] Current schedule: ! <Entry: celery.backendcleanup celery.backend_cleanup() <crontab: 0 4 * (m/h/d/dM/MY)> ! [2015-06-08 20:09:28,306: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes ! [2015-06-08 20:09:28,316: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2014 GoPivotal, Inc.', u'capabilities': {u'exchangeexchangebindings': True, u'connection.blocked': True, u'authenticationfailureclose': True, u'basic.nack': True, u'perconsumerqos': True, u'consumerpriorities': True, u'consumercancelnotify': True, u'publisherconfirms': True}, u'cluster_name': u'rabbit@administrators-MacBook-Air-4', u'platform': u'Erlang/OTP', u'version': u'3.4.4'}, mechanisms: [u'AMQPLAIN', u'PLAIN'], locales: [u'enUS'] ! [2015-06-08 20:09:28,317: DEBUG/MainProcess] Open OK! ! [2015-06-08 20:09:28,318: DEBUG/MainProcess] using channelid: 1 ! [2015-06-08 20:09:28,320: DEBUG/MainProcess] Channel open ! [2015-06-08 20:09:28,320: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,320: DEBUG/MainProcess] | Consumer: Starting Heart ! [2015-06-08 20:09:28,322: DEBUG/MainProcess] ^-- substep ok ! [2015-06-08 20:09:28,323: DEBUG/MainProcess] | Consumer: Starting Mingle ! [2015-06-08 20:09:28,323: INFO/MainProcess] mingle: searching for neighbors ! [2015-06-08 20:09:28,323: DEBUG/MainProcess] using channel_id: 1 ! [2015-06-08 20:09:28,325: DEBUG/MainProcess] Channel open ! [2015-06-08 20:09:28,332: DEBUG/Beat] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': HTTP/1.1 200 OK Transfer-Encoding: chunked Date: Thu, 21 Jul 2016 23:13:36 GMT Server: Warp/3.2.6 Content-Type: application/json

016A ["37zyefqi2sweveyp","42fn7zeo6v5ui427","66pt5sk2wz2jrbzu","awoljknjigytdyls","cj2lanoogknwopto","cnm3adnh35xmsx3f","ebxs4t2y6xr5izzy","eg5zus2pz72mr7xb","exshwew2w2jv3n7r","hxrxgzvgms3incmf","hymu5oh2f5ctk5jr","jkisbjnul226jria","lag7djeljbjng6bu","o3l65o4qzcxs327j","qsk2jzo2zh523r24","t7k6g7fkndoggutd","xfllvjyax4inadxh","ygtjzi2wkfonj3z7","yycjajwpguyno4je"] 0


User 19 | 6/8/2015, 8:12:22 PM

The file celeryapps.py includes import testceleryconfig. Is that another file that you have locally?


User 481 | 6/8/2015, 8:27:13 PM

Hi Chris, you can start the worker without celeryconfig file. I just edited the celeryapps.py file to do so. The config is most to define the messge broker, pre fetch rules, etc. not required for this trivial example.


User 19 | 6/8/2015, 9:38:57 PM

As a debugging step, can you try commenting out lines graphlab_util/init.py:23-63 wherever the egg is installed? If using virtualenv or something, it's typically in venv/lib/python2.7/site-packages/graphlab_util. Line 23 should be logging.config.dictConfig({... and the last line is logging.getLogger('librato').setLevel(logging.CRITICAL). These lines are aimed at making GraphLab Create a bit less verbose, but they may be causing issues with Celery.

Please try rerunning your example and let me know how it goes!


User 481 | 6/8/2015, 10:21:18 PM

Sweet, that worked! What was the issue, how is the connected to celery?


User 19 | 6/8/2015, 10:30:39 PM

Great!

We are still investigating the cause, but for some reason Celery hangs indefinitely(?) when a task contains logging.config.dictConfig() (where you have to first import logging and logging.config).


User 481 | 6/8/2015, 10:46:35 PM

http://celery.readthedocs.org/en/latest/configuration.html#logging Setting CELERYD_HIJACK_ROOT_LOGGER = False does not help either.