Reading and writing very large CSV files

User 2568 | 5/5/2016, 7:53:09 AM

In my current project I have to read and write very large CSV file to/from SFrames. While the current implementation works well, I wondered if you had looked as reading/writing compressed (gz) files. This might be much faster as my files are 10x compressed when gziped

Comments

User 2506 | 5/5/2016, 3:52:43 PM

gl.SFrame.read_csv should be able to take in a csv.gz file. Have you tried it?


User 2568 | 5/5/2016, 9:41:28 PM

Thanks... I tried it and graphlab can read and write csv.gz. Very handy!

Perhaps a line in the documentation to this effect.