Communication Failure: 65.

User 956 | 12/1/2014, 9:31:36 PM

Hey all,

I build an algorithm that needs some time (approx. 30 secs) to be executed. After some time I get the error: <code class="CodeInline">Communication Failure: 65.</code>

I attached the full log to this post.

I have a MacBook 8gb, i7 2 GHz, OSX 10.10 Yosemite. The script is running in virtualenv as well as ipython. The same error appears when I execute the script from the terminal (in virtualenv), too.

I hope someone has an idea what might be wrong.

Thanks for your help Tarek

Comments

User 10 | 12/1/2014, 10:53:27 PM

Can you share which version of GraphLab Create you are using? Is this on version 1.1?

Typically this error indicates that the C++ server process has crashed. In looking at the log file I don't see anything obvious, can you share a little bit more about what you are doing (feel free to private message me or email me at rajat@graphlab.com)?


User 10 | 12/2/2014, 7:08:12 PM

Thanks for sending me more details in email. I have confirmed that I can reproduce this error using GraphLab Create 1.1.

I will follow up with more information as we dig into this issue.


User 10 | 12/3/2014, 6:48:42 AM

<h2>Root Cause</h2> This issue is a bug in GraphLab Create 1.1 (and prior versions) <code class="CodeInline">SFrame.append</code> method. To encounter this bug, <code class="CodeInline">SFrame.append</code> must be executed many times on the same SFrames (most likely in a loop). The root cause of the bug is how SFrames are lazily evaluated, and the stack frame depth supported on different platforms (OS X has a much lower limit of stack frames than Linux). Due to this limit, if SFrames were appended (while not being fully materialized due to lazy evaluation) in a loop then it is possible to exhaust the number of stack frames for the system, causing the C++ process to crash. The C++ process crashing results in <code class="CodeInline">Communication Failure: 65.</code>

<h2>Fix</h2> We will fix <code class="CodeInline">SFrame.append</code> method in an upcoming release of GraphLab Create.

<h2>Workaround</h2> If you are experiencing this error and have many <code class="CodeInline">SFrame.append</code> operations in your code, the following is a workaround. Define a function <code class="CodeInline">safe_append</code> (as below), and use it instead of where your code currently calls <code class="CodeInline">SFrame.append</code>.

<pre class="CodeBlock"><code> def safeappend(sframeone, sframetwo): ''' Materializes SFrames prior to appending, ensures no stack frame exhaustion ''' sframeone.materialize() sframetwo.materialize() return sframeone.append(sframe_two) </code></pre>

This workaround will slow down the <code class="CodeInline">SFrame.append</code> operations because it requires full materialization of the SFrames (negating the performance benefits of lazy evaluation) but will prevent this bug from being encountered.


User 1296 | 2/19/2015, 6:15:14 PM

Hi,

A usage of the <code class="CodeInline">materialize()</code> method slows it down, but actually it is not necessary to call it before every <code class="CodeInline">SFrame.append</code> operation - this could save some time.

It was not fixed in the version 1.3.

Ondrej