RuntimeError: Communication Failure: 113. On topic_model.create with method='alias'

User 2032 | 9/29/2015, 10:09:42 AM

PROGRESS: Learning a topic model PROGRESS: Number of documents 3034687 PROGRESS: Vocabulary size 43480 PROGRESS: Initializing topic assignments PROGRESS: Constructed alias samplers:4.72587s PROGRESS: Using 128 blocks. PROGRESS: +-----------+---------------+----------------+-----------------+-------------------+ PROGRESS: | Iteration | Elapsed Time | Tokens/Second | Est. Perplexity | Elapsed for perp. | PROGRESS: +-----------+---------------+----------------+-----------------+-------------------+ PROGRESS: | 50 | 17m 33s | 1.05903e+06 | 0 | 0 | PROGRESS: | 100 | 34m 42s | 1.00083e+06 | 0 | 0 | PROGRESS: | 150 | 51m 55s | 1.05174e+06 | 0 | 0 | PROGRESS: | 200 | 1h 9m | 1.06093e+06 | 0 | 0 | PROGRESS: | 250 | 1h 25m | 1.11578e+06 | 0 | 0 |


RuntimeError Traceback (most recent call last) ...

This was on default options except the number of iterations was set to 1000. Setting and env as in my previous post.

Comments

User 19 | 9/29/2015, 1:57:01 PM

Thanks for the report. Were you able to reproduce this on a dataset that you could share? How many topics were you using?


User 2032 | 9/29/2015, 2:15:51 PM

I was using 100 topics. I cannot share the dataset. My current conjecture is that if there are multiple instances of GL a burst in memory could cause a OOM. I will try to reproduce while ensuring that there is only one gl server running on my machine.


User 2032 | 9/30/2015, 10:43:03 AM

Reproduced with single GL.

1443570110 : PROGRESS: (p:461): | 200 | 40m 18s | 1.43303e+06 | 0 | 0 | 1443570122 : INFO: (newcache:157): Cache Utilization:1190555503 1443570122 : INFO: (deletepathimpl:299): Deleting cache://tmp/001636.0000 1443570122 : INFO: (deletepathimpl:299): Deleting cache://tmp/001636.sidx 1443570134 : INFO: (newcache:157): Cache Utilization:1190555503 1443570134 : INFO: (deletepathimpl:299): Deleting cache://tmp/001637.0000 1443570134 : INFO: (deletepathimpl:299): Deleting cache://tmp/001637.sidx 1443570146 : INFO: (newcache:157): Cache Utilization:1190555503 1443570146 : INFO: (deletepathimpl:299): Deleting cache://tmp/001638.0000 1443570146 : INFO: (deletepathimpl:299): Deleting cache://tmp/001638.sidx 1443570158 : INFO: (newcache:157): Cache Utilization:1190555503 1443570158 : INFO: (deletepathimpl:299): Deleting cache://tmp/001639.0000 1443570158 : INFO: (deletepathimpl:299): Deleting cache://tmp/001639.sidx 1443570170 : INFO: (newcache:157): Cache Utilization:1190555503 1443570170 : INFO: (deletepathimpl:299): Deleting cache://tmp/001640.0000 1443570170 : INFO: (deletepathimpl:299): Deleting cache://tmp/001640.sidx 1443570182 : INFO: (newcache:157): Cache Utilization:1190555503 1443570182 : INFO: (deletepathimpl:299): Deleting cache://tmp/001641.0000 1443570182 : INFO: (deletepathimpl:299): Deleting cache://tmp/001641.sidx 1443570195 : INFO: (newcache:157): Cache Utilization:1190555503 1443570195 : INFO: (deletepathimpl:299): Deleting cache://tmp/001642.0000 1443570195 : INFO: (deletepathimpl:299): Deleting cache://tmp/001642.sidx 1443570206 : INFO: (newcache:157): Cache Utilization:1190555503 1443570206 : INFO: (deletepathimpl:299): Deleting cache://tmp/001643.0000 1443570206 : INFO: (deletepathimpl:299): Deleting cache://tmp/001643.sidx 1443570218 : INFO: (newcache:157): Cache Utilization:1190555503 1443570218 : INFO: (deletepathimpl:299): Deleting cache://tmp/001644.0000 1443570218 : INFO: (deletepathimpl:299): Deleting cache://tmp/001644.sidx 1443570230 : INFO: (newcache:157): Cache Utilization:1190555503 1443570230 : INFO: (deletepathimpl:299): Deleting cache://tmp/001645.0000 1443570230 : INFO: (deletepathimpl:299): Deleting cache://tmp/001645.sidx 1443570242 : INFO: (newcache:157): Cache Utilization:1190555503 1443570242 : INFO: (deletepathimpl:299): Deleting cache://tmp/001646.0000 1443570242 : INFO: (deletepathimpl:299): Deleting cache://tmp/001646.sidx 1443570254 : INFO: (newcache:157): Cache Utilization:1190555503 1443570254 : INFO: (deletepathimpl:299): Deleting cache://tmp/001647.0000 1443570254 : INFO: (deletepathimpl:299): Deleting cache://tmp/001647.sidx 1443570266 : INFO: (newcache:157): Cache Utilization:1190555503 1443570266 : INFO: (deletepathimpl:299): Deleting cache://tmp/001648.0000 1443570266 : INFO: (deletepathimpl:299): Deleting cache://tmp/001648.sidx 1443570278 : INFO: (newcache:157): Cache Utilization:1190555503 1443570278 : INFO: (deletepathimpl:299): Deleting cache://tmp/001649.0000 1443570278 : INFO: (deletepathimpl:299): Deleting cache://tmp/001649.sidx 1443570290 : INFO: (newcache:157): Cache Utilization:1190555503 1443570290 : INFO: (deletepathimpl:299): Deleting cache://tmp/001650.0000 1443570290 : INFO: (deletepathimpl:299): Deleting cache://tmp/001650.sidx 1443570302 : INFO: (newcache:157): Cache Utilization:1190555503 1443570302 : INFO: (deletepathimpl:299): Deleting cache://tmp/001651.0000 1443570302 : INFO: (deletepathimpl:299): Deleting cache://tmp/001651.sidx 1443570314 : INFO: (newcache:157): Cache Utilization:1190555503 1443570314 : INFO: (deletepathimpl:299): Deleting cache://tmp/001652.0000 1443570315 : INFO: (deletepathimpl:299): Deleting cache://tmp/001652.sidx 14HTTP/1.1 200 OK Transfer-Encoding: chunked Date: Thu, 21 Jul 2016 23:13:36 GMT Server: Warp/3.2.6 Content-Type: application/json

016A ["37zyefqi2sweveyp","42fn7zeo6v5ui427","66pt5sk2wz2jrbzu","awoljknjigytdyls","cj2lanoogknwopto","cnm3adnh35xmsx3f","ebxs4t2y6xr5izzy","eg5zus2pz72mr7xb","exshwew2w2jv3n7r","hxrxgzvgms3incmf","hymu5oh2f5ctk5jr","jkisbjnul226jria","lag7djeljbjng6bu","o3l65o4qzcxs327j","qsk2jzo2zh523r24","t7k6g7fkndoggutd","xfllvjyax4inadxh","ygtjzi2wkfonj3z7","yycjajwpguyno4je"] 0


User 2032 | 9/30/2015, 10:43:10 AM

1443570458 : INFO: (newcache:157): Cache Utilization:1190555503 1443570458 : INFO: (deletepathimpl:299): Deleting cache://tmp/001664.0000 1443570458 : INFO: (deletepathimpl:299): Deleting cache://tmp/001664.sidx 1443570470 : INFO: (newcache:157): Cache Utilization:1190555503 1443570470 : INFO: (deletepathimpl:299): Deleting cache://tmp/001665.0000 1443570470 : INFO: (deletepathimpl:299): Deleting cache://tmp/001665.sidx 1443570481 : INFO: (newcache:157): Cache Utilization:1190555503 1443570481 : INFO: (deletepathimpl:299): Deleting cache://tmp/001666.0000 1443570481 : INFO: (deletepathimpl:299): Deleting cache://tmp/001666.sidx 1443570493 : INFO: (newcache:157): Cache Utilization:1190555503 1443570493 : INFO: (deletepathimpl:299): Deleting cache://tmp/001667.0000 1443570493 : INFO: (deletepathimpl:299): Deleting cache://tmp/001667.sidx 1443570505 : INFO: (newcache:157): Cache Utilization:1190555503 1443570505 : INFO: (deletepathimpl:299): Deleting cache://tmp/001668.0000 1443570505 : INFO: (deletepathimpl:299): Deleting cache://tmp/001668.sidx 1443570517 : INFO: (newcache:157): Cache Utilization:1190555503 1443570517 : INFO: (deletepathimpl:299): Deleting cache://tmp/001669.0000 1443570517 : INFO: (deletepathimpl:299): Deleting cache://tmp/001669.sidx < ----------- crash with 113