recommender single machine and multipule cores

User 1912 | 5/11/2015, 10:30:36 AM

hi, I couldn't find this information on the website so I am just asking it here. Is the recommender toolkit already use multiple cores on a single machine? if not, is it possible to do that?

Thanks, Mohsen


User 91 | 5/11/2015, 2:27:59 PM

Yes, that is right, the recommender toolkit already uses all the cores on your machine.

User 2032 | 6/15/2015, 9:59:38 AM

Can you limit the number of cores it is allowed to use?

Use cases:

  1. Production machine runs GL create but also has other responsibilities.
  2. Running multiple GL create jobs in parallel on a single machine.

I assume it should be somwhere under graphlab.setruntimeconfig but where is this documented? I couldn't find it using search on readthedocs and looking through the source also did not yield any results regarding CPU.

User 19 | 6/15/2015, 8:21:02 PM

Hi Johnny,

Unfortunately, setting the number of threads is not currently possible via setruntimeconfig. As shown by your use cases, this is a good feature request! We will look into it.

Out of curiosity, how many cores do you typically have available?

Cheers, Chris

User 2032 | 6/15/2015, 8:31:30 PM

Hi Chris,

I'm surprised this is not available, seems like a critical feature for production use in my world, hope you guys can deliver it soon. Maybe in a minor release. This would also be nice for local usage because even my music freezes when I'm running topic modeling:)

On my laptop I have 4 cores 8 threads, in production GL will either run on 24 threads or 56. Once we grow I was planning to give it 128 cores if I see the need - but when I figured out the topic models perfomance I don't think this will be nessesary.

Kind regards, Jan

User 1207 | 6/15/2015, 8:36:25 PM

Hi Johnny,

It is actually easy to do what you want in your context. While it can't be changed at runtime, it's easy to explicitly set the number of cores used with the OMPNUMTHREADS environment variable. In bash, setting export OMP_NUM_THREADS=8 will tell GLC to run with 8 threads. -- Hoyt

User 2032 | 6/15/2015, 8:38:35 PM

Good stuff, Hoyt. Cheers! This helps a lot - I was already planning for separate virtual machines on our host...