What does this mean: graphlab.toolkits._main.ToolkitError: Fail to write. Disk may be full.

User 1761 | 4/13/2015, 8:58:39 PM

Hi,

I am trying to build a collaborative filter on a data set in a file of size 6.7GB. I have spun up an r3.4xlarge instance on AWS to accomplish this. However, when I run this line:

<pre class="CodeBlock"><code>gl.recommender.create(data, userid='X1', itemid='X2', target='X3') </code></pre>

It reports:

<pre class="CodeBlock"><code> File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/recommender/util.py", line 188, in create return graphlab.recommender.rankingfactorizationrecommender.create(**opts) File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/recommender/rankingfactorizationrecommender.py", line 267, in create response = graphlab.toolkits.main.run('recsystrain', opts, verbose) File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/main.py", line 99, in run raise ToolkitError(str(message)) graphlab.toolkits._main.ToolkitError: Fail to write. Disk may be full. </code></pre>

df -h reports plenty of free space:

<pre class="CodeBlock"><code>ubuntu@ip-10-51-XX:/dev/mahout$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.8G 1.6G 5.8G 22% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 61G 6.7G 54G 12% /dev tmpfs 13G 344K 13G 1% /run none 5.0M 0 5.0M 0% /run/lock none 61G 0 61G 0% /run/shm none 100M 0 100M 0% /run/user </code></pre>

Note that the dev drive has 54 GB free.

What is the cause of this error and how do I get around it? Thanks!

Comments

User 1394 | 4/13/2015, 9:23:59 PM

Hey esilver -

Thanks for getting started with GraphLab Create. The root partition on this EC2 instance (and most EC2 instances by default) is 8GB. The 54GB /dev partition is not a real partition - it doesn't represent actual disk space.

Depending on the instance type, the solution here is to format and mount the instance storage on the instance. Typically one of the instance storage devices is <code class="CodeInline">/dev/xvdb</code>. Below is some pseudo code for how to format and mount this device from a shell prompt. FYI, GraphLab Create does this for you if using the <a href="https://dato.com/products/create/docs/generated/graphlab.aws.launchEC2.html#graphlab.aws.launchEC2">graphlab.aws.launch_ec2</a> API to have your local Python session be powered by an EC2 instance.

Also, for a 6.7GB dataset, you shouldn't need a giant instance (r3.4xlarge). With GraphLab Create the data doesn't have to fit into RAM, the SFrame is an out-of-core data structure that spills to disk. For 7GB dataset you could comfortably use an m3.xlarge instance. The biggest difference is how quickly the models will be trained (number of cores and amount of disk IO).

Let me know if this helps.

Thanks,

Rajat

Example shell script to run for formatting and mounting one of the instance stores (these commands work for Ubuntu-based Linux AMIs on EC2):

<pre class="CodeBlock"><code>sudo mkfs.ext4 /dev/xvdb sudo mount /dev/xvdb sudo chmod +1777 /mnt sudo mkdir -m 1777 /mnt/tmp sudo mkdir -m 1777 /mnt/vartmp sudo mount --bind /mnt/tmp /tmp sudo mount --bind /mnt/vartmp /var/tmp</code></pre>


User 1761 | 4/13/2015, 10:01:26 PM

Thanks for the info. I still seem to be hitting this issue after switching to an HVM m3.xlarge, which comes with /dev/xvdb already mounted for you. Here's the error:

<pre class="CodeBlock"><code>PROGRESS: Recsys training: model = rankingfactorizationrecommender PROGRESS: Preparing data set. [ERROR] Toolkit error: Fail to write. Disk may be full. Traceback (most recent call last): File "filter.py", line 5, in <module> model = gl.recommender.create(data, userid='X1', itemid='X2', target='X3') File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/recommender/util.py", line 188, in create return graphlab.recommender.rankingfactorizationrecommender.create(**opts) File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/recommender/rankingfactorizationrecommender.py", line 267, in create response = graphlab.toolkits.main.run('recsystrain', opts, verbose) File "/usr/local/lib/python2.7/dist-packages/graphlab/toolkits/main.py", line 99, in run raise ToolkitError(str(message)) graphlab.toolkits._main.ToolkitError: Fail to write. Disk may be full. </code></pre>

And here is the df -h:

<pre class="CodeBlock"><code>ubuntu@ip-10-169-XXX:/mnt/graphlab$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.8G 1.6G 5.8G 22% / none 4.0K 0 4.0K 0% /sys/fs/cgroup udev 7.4G 12K 7.4G 1% /dev tmpfs 1.5G 204K 1.5G 1% /run none 5.0M 0 5.0M 0% /run/lock none 7.4G 0 7.4G 0% /run/shm none 100M 0 100M 0% /run/user /dev/xvdb 37G 6.8G 29G 20% /mnt </code></pre>

As you can see, /dev/xvdb has 29GB free, and is storing my 6.7GB file.

Is it possible that graphlab is still pointing to the root volume, even though I am running my python script and directing its output to my /dev volume? Thanks!


User 1761 | 4/13/2015, 10:29:40 PM

As I run my script, I can see the utilization of /dev/xvda1 creeping up. This appears to be due to the fact that graphlab is writing large files to /var/tmp/graphlab-ubuntu

I have therefore just rmdir'd /var/tmp/graphlab-ubuntu and created a symbolic link from /var/tmp/graphlab-ubuntu to a folder I created at /mnt/graphlab/tmp.

Is there a better way to do this, perhaps by specifying the tmp folder to use at initialization time?


User 1394 | 4/13/2015, 10:59:56 PM

Yep, the following lines do the mounting, and can be done on startup:

<pre class="CodeBlock"><code>sudo chmod +1777 /mnt sudo mkdir -m 1777 /mnt/tmp sudo mkdir -m 1777 /mnt/vartmp sudo mount --bind /mnt/tmp /tmp sudo mount --bind /mnt/vartmp /var/tmp </code></pre> This way all the /tmp and /var/tmp files are actually coming from /dev/xvdb.

Thanks,

Rajat


User 1761 | 4/13/2015, 11:38:32 PM

Right, of course. Everything seems to be running now. Thanks so much, and thank you for the suggestion that I could use a smaller instance type! Much more savings over running the massive r3 types.

(I started with r3 because mahout required significant amounts of processing power to handle the data, but I am hopeful that graphlab will be much more efficient in its use of resources.)


User 1394 | 4/14/2015, 1:55:12 AM

Glad to read things are working now.

With 7GB dataset, you can probably do that on your laptop with GraphLab Create, but it will be faster on a beefier EC2 instance.

On a basic Macbook Pro (8GB RAM, 4-core) we have easily processed 20GB datasets, so you might not need to spend any money on EC2 to get started on this dataset.

Thanks,

Rajat