errors trying to create an ec2 instance via Ec2Config and ec2_cluster

User 2785 | 12/11/2015, 5:54:21 PM

We're trying to run graphlab on AWS and have been unable to create a data distributed cluster. This is the code we're running:

`python import graphlab as gl

myconfig = gl.deploy.Ec2Config(awsaccesskeyid=<keyid>, awssecretaccesskey=<secretaccesskey>,region=<region>,securitygroupid=<securitygroupid>) mycluster = gl.deploy.ec2cluster.create(name=<name>,s3path=<bucket>,ec2config=my_config)

my_cluster.stop() ` and we get back this error:

` [INFO] 1449854473 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/certifi/cacert.pem 1449854473 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERT_DIR to This trial license of GraphLab Create is assigned to REDACTED and will expire on December 20, 2015. Please contact trial@dato.com for licensing options or to request a free non-commercial license for personal or academic use.

[INFO] Start server at: ipc:///tmp/graphlabserver-629 - Server binary: /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/unityserver - Server log: /tmp/graphlabserver1449854473.log [INFO] GraphLab Server Version: 1.7.1 [INFO] 1449854474 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/certifi/cacert.pem 1449854474 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTDIR to This trial license of GraphLab Create is assigned to REDACTED and will expire on December 20, 2015. Please contact trial@dato.com for licensing options or to request a free non-commercial license for personal or academic use.

Traceback (most recent call last): File "tryawsstuffs.py", line 17, in <module> ec2config=myconfig File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/ec2cluster.py", line 83, in create cluster.start() File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/ec2cluster.py", line 233, in start self.idleshutdowntimeout File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/executionenvironment.py", line 355, in startcommanderhost producttype = ProductType.DatoDistributed) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 420, in ec2factory subnetid = subnetid, securitygroupid = securitygroupid) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 365, in setupsecuritygroup securitygroupname, securitygroupid, subnetid) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 281, in getsecuritygroupconfig subnetid = subnets[0].id IndexError: list index out of range [INFO] Stopping the server connection. ` any thoughts as to what might be going on here? Thanks!

Comments

User 15 | 12/11/2015, 6:56:37 PM

Hmm...that looks like a bug at least from the standpoint that we should be checking for a delivering a proper error message. Looking at the code it looks like we're searching for subnets with the VPC ID you provided to securitygroupid, and the connection to AWS is returning no subnets, but our code assumes there will be at least one. Could you try providing the name of the security group as well?

Evan


User 2785 | 12/11/2015, 6:59:38 PM

Hi Evan,

The securitygroupid name is sg-9bf32ef0

I've tried running the same script but without the securitygroupid and i get the same error.


User 15 | 12/11/2015, 7:06:09 PM

I meant to provide the name to the Ec2Config for the security_group parameter. Did you try that?


User 2785 | 12/11/2015, 7:14:53 PM

the security group is called 'ElasticMapReduce-master' according to looking up currently running instances our bucket's EMR. When I change the Ec2Config to the following:

python my_config = gl.deploy.Ec2Config( aws_access_key_id=<id>, aws_secret_access_key=<key>, region='us-east-1', security_group='ElasticMapReduce-master' )

i get the following error:

`[INFO] 1449861119 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/certifi/cacert.pem 1449861119 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERT_DIR to This trial license of GraphLab Create is assigned to elizabeth.orr@urbanairship.com and will expire on December 20, 2015. Please contact trial@dato.com for licensing options or to request a free non-commercial license for personal or academic use.

[INFO] Start server at: ipc:///tmp/graphlabserver-2686 - Server binary: /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/unityserver - Server log: /tmp/graphlabserver1449861119.log [INFO] GraphLab Server Version: 1.7.1 [INFO] 1449861121 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to /Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/certifi/cacert.pem 1449861121 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTDIR to This trial license of GraphLab Create is assigned to elizabeth.orr@urbanairship.com and will expire on December 20, 2015. Please contact trial@dato.com for licensing options or to request a free non-commercial license for personal or academic use.

Traceback (most recent call last): File "tryawsstuffs.py", line 19, in <module> ec2config=myconfig File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/ec2cluster.py", line 83, in create cluster.start() File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/ec2cluster.py", line 233, in start self.idleshutdowntimeout File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/deploy/executionenvironment.py", line 355, in startcommanderhost producttype = ProductType.DatoDistributed) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 420, in ec2factory subnetid = subnetid, securitygroupid = securitygroupid) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 365, in setupsecuritygroup securitygroupname, securitygroupid, subnetid) File "/Applications/anaconda/envs/CHURNSTUFFS/lib/python2.7/site-packages/graphlab/connect/aws/ec2.py", line 333, in getsecuritygroupconfig raise Exception('Error: No Subnet inside VPC ' + str(securitygroup.vpc_id)) Exception: Error: No Subnet inside VPC None [INFO] Stopping the server connection.`


User 2785 | 12/11/2015, 7:27:56 PM

I see that our security group does not have an associated VPC, is that an issue with creating a data distributed cluster?


User 16 | 12/11/2015, 8:48:32 PM

Hi Wallawalla-

I'm sorry you're experiencing this issue.

I agree with Evan that our code should not be assuming we'll always get back a subnet id from our AWS security config call. I'll make sure this assumption is fixed in our next release.

In the mean time, I would suggest not specifying a securitygroupid in your Ec2Config (in which case we'll automatically create a security group for you). This is how we typically create a cluster. Please try that.

Thanks, Toby