create SFrame from numpy array of size~ 1GB. memory overflows and computer need reset key

User 2962 | 1/7/2016, 6:01:59 PM

Hey

I do the following

import numpy as np import graphlab as gl x=np.random.rand(1500000,250) test=gl.SFrame({'x': x})

I am running windows 7 with 16GB and running python terminal

The usage of memory grows up fast and system halts totally. Forced reset reboot needed I tried few times, once there was an assertion error of buffer size not big enough and the terminal just crashed but most of the time just becomes non responsive

Comments

User 940 | 1/7/2016, 7:32:14 PM

Hi @didist ,

Could you try loading it in batch? Something like this:

python import numpy as np import graphlab as gl x=np.random.rand(1500000,250) test_1=gl.SFrame({'x': x[0:750000]}) test_2=gl.SFrame({'x': x[750000:]}) test_final = test_1.append(test_2)

Let me know if this works!

Cheers! -Piotr


User 2962 | 1/8/2016, 1:36:28 AM

Thanks

It works now

Why does the memory usage in SFrame creation process so high, but then it frees the non needed data Does it have to do with me using numpy in windows? Is there any other better way to load larger data files simply without batch loading?