Communication Failure: 65

User 2263 | 12/3/2015, 3:10:10 PM

Hi Nice People,

Since yesterday I'm getting a Communication Failure: 65 error every time I try to create/learn a model from my data. My code looks like:

` In [1]: import graphlab as gl

In [2]: od = gl.loadsframe('/thepath/userlikebyartistsframe') [INFO] 1449152559 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTFILE to /usr/local/lib/python2.7/site-packages/certifi/cacert.pem 1449152559 : INFO: (initializeglobalsfromenvironment:282): Setting configuration variable GRAPHLABFILEIOALTERNATIVESSLCERTDIR to This non-commercial license of GraphLab Create is assigned to me@thisplace.com and will expire on September 11, 2016. For commercial licensing options, visit https://dato.com/buy/. [INFO] Start server at: ipc:///tmp/graphlabserver-809 - Server binary: /usr/local/lib/python2.7/site-packages/graphlab/unityserver - Server log: /tmp/graphlabserver_1449152559.log [INFO] GraphLab Server Version: 1.7.1

In [3]: model = gl.factorizationrecommender.create( observationdata=od, userid='user', itemid='mbid', target='likert', numfactors=2, maxiterations=10, regularization=1e-07) PROGRESS: Recsys training: model = factorization_recommender PROGRESS: Preparing data set. `

After some time I'm always getting the error

` PROGRESS: Recsys training: model = factorization_recommender PROGRESS: Preparing data set.


RuntimeError Traceback (most recent call last) <ipython-input-3-6aecd526de4a> in <module>() 6 numfactors=2, 7 maxiterations=10, ----> 8 regularization=1e-07)

/usr/local/lib/python2.7/site-packages/graphlab/toolkits/recommender/factorizationrecommender.pyc in create(observationdata, userid, itemid, target, userdata, itemdata, numfactors, regularization, linearregularization, sidedatafactorization, nmf, binarytarget, maxiterations, sgdstepsize, randomseed, solver, verbose, **kwargs) 230 opts.update(kwargs) 231 --> 232 response = graphlab.toolkits.main.run('recsystrain', opts, verbose) 233 234 return FactorizationRecommender(response['model'])

/usr/local/lib/python2.7/site-packages/graphlab/toolkits/main.pyc in run(toolkitname, options, verbose, showprogress) 58 try: 59 starttime = time.time() ---> 60 (success, message, params) = unity.runtoolkit(toolkitname, options) 61 end_time = time.time() 62 except:

graphlab/cython/cyunity.pyx in graphlab.cython.cyunity.UnityGlobalProxy.run_toolkit()

graphlab/cython/cyunity.pyx in graphlab.cython.cyunity.UnityGlobalProxy.run_toolkit()

RuntimeError: Communication Failure: 65. `

I have run the code in two machines running OSX (10.10.5, and 10.9.5) with different configurations of RAM and CPU/Cores with same results. There are no warnings or errors in the log file.

Any clues?

Comments

User 1207 | 12/3/2015, 6:12:03 PM

Hello Nice Person,

I'm sorry you're having trouble with this. How large is your data, how many columns, and what are the types of the columns? How many unique users and unique items? Knowing this will help me get a handle on tracking this down.

You can get the number of unique users and items with:

print od["user"].unique().size() print od["mbid"].unique().size()

-- Hoyt


User 2263 | 12/3/2015, 7:03:47 PM

Hi @hoytak ,

For now, the data I'm processing has 675629343 observations with 593725 unique users and 555214 unique items. The SFrame has five columns with the following types: [str, int, str, int, int]


User 1207 | 12/3/2015, 7:44:51 PM

Hello @Vigliensoni,

Thanks; that is helpful.

Next question -- are the user and item columns the two string columns in your data, or are those side columns? The reason I ask is that string columns can take more memory due to the indexing involved, and I'm wondering if that is an issue in your case. If not, how many unique values does the side column of type string have?

Also, could you try dropping everything but the user, item, and target column and see if it works with that?

Thanks for your patience. We'll get to the root of the issue.

-- Hoyt


User 2263 | 12/4/2015, 1:10:52 PM

Good morning @hoytak

I drop all other columns but {user, item, target}, with same results. have tried the default values for the configuration variables GRAPHLAB_FILEIO_MAXIMUM_CACHE_CAPACITY (2GB) and GRAPHLAB_FILEIO_MAXIMUM_CACHE_CAPACITY_PER_FILE (2GB), but I also have tried (8GB, 512MB) to try to maximize the performance of my system.

Any clues?


User 1207 | 12/10/2015, 10:37:29 PM

Hello Vigliensoni,

Thanks for your patience; sorry to not get back to you before now. I'm not sure what is happening, although I suspect that something is unexpectedly running out of memory. Is there any way you could share your data with us? If so, I will look into it and try to reproduce the issue. Feel free to email support@dato.com or send me a personal message, and I'll send you details there.

Otherwise, one thing to do to trim memory footprint here is to clean out users or items that have less than, say, 3 or 5 ratings. In this case.

-- Hoyt