Creating one's own dataset of images for image analysis?

User 2403 | 10/13/2015, 10:06:41 PM

Hello, can someone point me to a tutorial or instructions on how to create / import my own data set of images for analysis in GraphLab?

Thanks!

Comments

User 940 | 10/14/2015, 6:32:33 PM

Hi @chumann ,

Here's a set of API docs that loads in a set of images https://dato.com/products/create/docs/generated/graphlab.imageanalysis.loadimages.html.

Does this help?

Cheers! -Piotr


User 2403 | 10/14/2015, 11:29:31 PM

Thank you for your response Piotr. I tried what you suggested in the referenced API and get the following error which I'm having troubles deciphering (not sure what "AttributeError: 'Image' object has no attribute 'replace'" is telling me):

In [1]:import graphlab In [57]:img = graphlab.Image('imagetraindatacloud/') In [65]:imagesarray = graphlab.imageanalysis.loadimages(img, "JPG", with_path=False,

... recursive=True,ignore_failure=True)


AttributeError Traceback (most recent call last) <ipython-input-65-07f63e6fc477> in <module>() 1 imagesarray = graphlab.imageanalysis.loadimages(img, "JPG", withpath=False, ----> 2 recursive=True,ignore_failure=True)

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/toolkits/imageanalysis/imageanalysis.pyc in loadimages(url, format, withpath, recursive, ignorefailure, randomorder) 59 from ... import extensions as extensions 60 from ...util import makeinternalurl ---> 61 return extensions.loadimages(makeinternalurl(url), format, withpath, 62 recursive, ignorefailure, randomorder) 63

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/util/init.pyc in makeinternalurl(url) 181 182 # Convert Windows paths to Unix-style slashes --> 183 url = convert_slashes(url) 184 185 # The final file path on server.

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/util/init.pyc in convertslashes(path) 99 Converts all windows-style slashes to unix-style slashes 100 """ --> 101 return path.replace('\', '/') 102 103 def getaws_credentials():

AttributeError: 'Image' object has no attribute 'replace'


User 940 | 10/14/2015, 11:54:27 PM

Hi @chumann,

It does look like you're doing things correctly, and that this is a bug. Three questions:

  1. Does the same problem occur when you run

`python

url ='s3://gl-testdata/images/nested' imagesarray = graphlab.imageanalysis.loadimages(url, "auto", withpath=False, ... recursive=True) ` 2. When your run your first GraphLab command, there should be some output saying where the server log is. Could you share the output of that?

  1. What kind of System is this on? Which OS?

This should help us with debugging.

Cheers! -Piotr


User 2403 | 10/15/2015, 12:43:56 AM

Apologies Piotr, I tried your solution in haste and forgot to change the first line of code. It seems to work now. Thank you!


User 2403 | 10/15/2015, 12:58:06 AM

Hi Piotr, As mentioned earlier, I think I resolved my initial problems due to syntax errors on the second line of code where I had:

img = graphlab.Image('imagetraindatacloud/') and it should have been: img = ('imagetraindatacloud/')

To answer your last questions: I'm using OSX 10.9.5 The server log listed on output is : INFO] Start server at: ipc:///tmp/graphlabserver-6367 - Server binary: /Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/unityserver - Server log: /tmp/graphlabserver1444870035.log [INFO] GraphLab Server Version: 1.6.1

Out of curiosity I tried running your last suggestion and received the following error:

url ='s3://gl-testdata/images/nested' imagesarray = graphlab.imageanalysis.loadimages(url, "auto", withpath=False, ... recursive=True) KeyError Traceback (most recent call last) <ipython-input-3-6b9b4b89385a> in <module>() 1 imagesarray = graphlab.imageanalysis.loadimages(url, "auto", withpath=False, ----> 2 recursive=True)

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/toolkits/imageanalysis/imageanalysis.pyc in loadimages(url, format, withpath, recursive, ignorefailure, randomorder) 59 from ... import extensions as extensions 60 from ...util import makeinternalurl ---> 61 return extensions.loadimages(makeinternalurl(url), format, withpath, 62 recursive, ignorefailure, randomorder) 63

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/util/init.pyc in makeinternalurl(url) 204 return url 205 elif protocol == 's3': --> 206 return tryinjects3_credentials(url) 207 elif protocol == 'remote': 208 # url for files on the server

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/util/init.pyc in tryinjects3credentials(url) 148 149 # S3 url does not contain secret key/id pair, query the environment variables --> 150 (k, v) = getaws_credentials() 151 return 's3://' + k + ':' + v + ':' + path 152

/Users/chumann/.graphlab/anaconda/lib/python2.7/site-packages/graphlab/util/init.pyc in getawscredentials() 125 126 if (not 'AWSACCESSKEYID' in os.environ): --> 127 raise KeyError('No access key found. Please set the environment variable AWSACCESSKEYID, or using graphlab.aws.setcredentials()') 128 if (not 'AWSSECRETACCESSKEY' in os.environ): 129 raise KeyError('No secret key found. Please set the environment variable AWSSECRETACCESSKEY, or using graphlab.aws.set_credentials()')

KeyError: 'No access key found. Please set the environment variable AWSACCESSKEYID, or using graphlab.aws.setcredentials()'


User 2403 | 10/15/2015, 5:16:26 PM

Hello again Piotr,

Could you point me to a tutorial / example on how to work with one's own data set of images? I'm trying to figure out how to prepare images to serve as a learning set. I'm under the assumption the each image needs some sort of 'label' included appended to its meta data?


User 940 | 10/16/2015, 5:40:56 PM

Hi @chumann ,

Good catch! But really, our product should be giving more clear error messages.

In general, yes, you do need some label for each image. This label is typically appended manually. How you do this, however, is specific to the task at hand. I can take a look around to try and find such a tutorial though.

What are you trying to accomplish in this case?

Cheers! -Piotr


User 2403 | 10/18/2015, 5:45:09 PM

Hi Piotr, Thanks once again for the follow up. I have a series of sky images, some representing overcast sky conditions, others showing intermediate and clear conditions. I would like to prepare a learning set such that I can eventually automatically identify 'sky conditions' of new images by passing them through a neural network.

Cheers, Chris