launch_EC2 later followed by model_parameter_search

User 1375 | 3/9/2015, 7:55:09 PM

I've successfully launched a GraphLab Server in EC2 via launchEC2, loaded some data from S3 into that server, done some feature engineering, trained a default logistic regression classifier, and now I'm ready to calibrate, say, the regularization hyperparameter l2penlaty using modelparametersearch. I'm finding that there's no way to use that API function in a way that makes use of the existing EC2 instance running the GraphLab Server. Is that correct? I've tried environment.Local and environment.AsyncLocal, but those seem to refer literally to my local ipython process outside of EC2, instead of "local" from the point of view of the GraphLab Server running in EC2, which was another possible interpretation. I was looking to perform this parameter search within the existing GraphLab Server so as not to have to move the data around i.e. copy to S3, spin up new EC2 machines via modelparametersearch feeding it environment.EC2(...), etc. I just wanted to exploit the various vCPUs in the existing GraphLab Server, and thought that environment.LocalAsync would do it. Any help would be highly appreciated. Thanks.

Marcos

Comments

User 18 | 3/9/2015, 10:21:24 PM

Hi Marcos,

Right now modelparametersearch (and Jobs in general) behaves rather awkwardly when you are running remotely on an EC2 instance. You are correct that it will not use the current instance but rather spin up new EC2 instances to do the work. modelparametersearch returns a Jobs object that will contain the results of the parameter search once it's done. While the search is going on, you'd have to keep the original EC2 instance running in order to retain the handle on that Job object.

This is slightly convoluted and we're actively seeking workarounds.

For now, there's two options:

  1. If you can keep the original EC2 instance, then everything still works. Modelparametersearch will automatically save to S3. So the only cost is saving out the data to S3 and keeping an extra instance running.

  2. You can start modelparametersearch job on EC2 from your local machine. That way the monitoring is free. To do this, first <a href="https://dato.com/products/create/docs/generated/graphlab.deploy.environment.EC2.html#graphlab.deploy.environment.EC2">create an EC2 environment</a>. You can specify the S3 path and numhosts. Then, pass this environment to <a href="https://dato.com/products/create/docs/graphlab.toolkits.modelparametersearch.html">modelparameter_search</a> via the 'environment' option.

Let us know how it goes.

Alice


User 1319 | 3/10/2015, 12:17:25 AM

Hi @Alice,

I'm running a modelparametersearch to optimize SEVERAL hyperparameters of a Boosted Tree model. The search is currently being executed on an EC2's GPU instance.

Based on the job's outputs, it seems the modelparametersearch is done sequentially.

Does the modelparametersearch support parallel execution on a GPU?

Thanks,

Tarek


User 91 | 3/10/2015, 3:38:05 AM

We currently do not support parallel execution on a single GPU instance but we do support parallel execution on a multi-node i.e distributed setting.

You can do this by setting the num_hosts parameter in the environment variable. This will spin up multiple nodes and distribute the work across each of the nodes.


User 1319 | 3/10/2015, 3:59:45 AM

Thanks Srikris for your quick reply.

Just to clarify, the modelparametersearch also does not support multi-core execution on the same EC2 instance (say an EC2 with 32 cores)?

Note: I'm running GraphLab Create directly on the EC2 instance via IPython Server (to avoid moving big data files).

Tarek


User 16 | 3/10/2015, 5:20:09 PM

Our Engine does use all available cores. So modelparametersearch will do multi-core execution, it's just that it will only do one model at a time. If you want more than one model to be trained and evaluated in parallel, you can use more than one EC2 host.


User 1319 | 3/10/2015, 5:28:34 PM

Thank you Toby for the clarification. Highly appreciated. Tarek