Creating Predictive Service Times out

User 3122 | 2/6/2016, 9:48:19 AM

What am I doing wrong here? I followed the tutorial here and it never finishes - it gets stuck on step four just repeatedly printing:

[INFO] Cluster not fully operational yet, [0/1] instances currently in service.

Here's my python code. It seems to work fine as it launches the instance, the load balancer, the security group, and I see a license and state.ini file added to the s3 bucket. But my ipython notebook just never returns anything?

And I'm using my root aws access key so I assume it has full rights. Any idea what I'm doing wrong?

import graphlab
env = graphlab.deploy.Ec2Config(region='us-west-2',
                                instance_type='g2.2xlarge',
                                aws_access_key_id='xxxxxxxx', 
                                aws_secret_access_key='xxxxxxxxxxx')
deployment = graphlab.deploy.predictive_service.create('service-test-two', env, 's3://my-bucket-name/service-test-two')
deployment.save()

Comments

User 2506 | 2/8/2016, 12:57:08 AM

Hi Chipa,

Let me help you to figure out what went wrong. How long did you wait when you see this message "[INFO] Cluster not fully operational yet, [0/1] instances currently in service"?

The same message would be repeated until the instance is up and running, and it might take up to 10 minutes. If it doesn't work, you should see a timeout error message. Did you see the time out message?


User 2506 | 2/8/2016, 3:55:37 AM

A few other questions that will help us to troubleshoot your problems.

1) Can you tell us the version you are using? You can run the following commands. import graphlab as gl gl.version

2) Are you using EC2-VPC or EC2-Classic? Refer to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html for details.


User 5249 | 5/30/2016, 10:09:05 AM

I am experiencing the same issue. I am using graphlab version 1.9 and our aws account supports both EC2-VPC and EC2-Classic.

I am calling: ec2 = gl.deploy.Ec2Config(region='eu-west-1', instancetype='m3.large', awsaccesskeyid='xxxx', awssecretaccesskey='xxxx') deployment = gl.deploy.predictiveservice.create('first', ec2, 's3://my-bucket-name/dato-predictive-services')

Everything looks okay, up to and including this message: [INFO] graphlab.deploy.predictive_service, 276: [Step 4/5]: Waiting for Load Balancer to put all instances into service.

But then for 30 minutes I get: [INFO] graphlab.deploy.predictive_service, 289: Cluster not fully operational yet, [0/1] instances currently in service.

And then I get the errors: [ERROR] graphlab.deploy.predictive_service, 292: Instances failed to be ready within 10 minutes. Tearing down. RuntimeError: Cluster configuration not successful in time, timing out.

Is there a way that I can get more detailed error information? Why might the instance be failing to be ready? Thank you.


User 5249 | 5/31/2016, 10:10:59 PM

For anyone else who runs into this problem, I was able to successfully create a predictive service deployment once I explicitly set subnet_id on the Ec2Config (https://dato.com/learn/userguide/deployment/pred-launching.html#nondefault-virtual-private-cloud).


User 5393 | 7/19/2016, 12:25:06 PM

I am experiencing similar issue waited for 40 min but still stuck on [Step 4/5]: Waiting for Load Balancer to put all instances into service. INFO Cluster not fully operational yet, [0/1] instances currently in service

As suggested by mist02468 i explicitly set subnet_id on the Ec2Config but still the same problem.