SFrame, SGraph and SArray cannot be pickled

User 1129 | 1/22/2015, 9:48:52 AM

GraphLab object cannot be pickled using the standard pickle modules. As the result, objects that either contain SGraph, SArray or SFrame data members or are derived from these classes cannot be pickled either.

Comments

User 15 | 1/22/2015, 6:52:45 PM

Hi,

Since all of these data structures are not Python objects, but stored in the C++ engine, it's not currently possible for you to pickle them. Due to this complexity, there are no plans to expose the functionality of pickling these data structures through the standard Python pickle package.

A fairly simple workaround is to save these objects to the filesystem, and keep pointers to the paths within your class that contains these. On init, your class can simply load from the file paths.

This won't help you when writing classes derived from these objects though. We didn't design these data structures to be extended in this way, and this is the first instance I've heard of someone trying to do this. So if you don't mind, I'd be interested to hear the use case for why you are interested in this.

Thanks,

Evan


User 1129 | 1/23/2015, 8:30:51 AM

Regarding using pointers to the filesystem: sounds like a reasonable workaround. Regarding derived classes: I don't have any particular use case. I noticed this behaviour when trying to pickle an object that contained an SFrame, then I started testing other things in order to make this report as wide as possible.

Thank you for the answers.


User 91 | 2/24/2015, 4:45:34 PM

You can use the Graphlab pickler (released in GLC 1.3) to pickle python and Graphlab objects. The pickler supports S3 and HDFS uploads.

    from graphlab.util import gl_pickle
    import graphlab as gl

    obj = {'model-1': model_1,
               'model-2': model_2,
               'data: gl.SArray([1,3,4])}

    # Setup the GLC pickler
    pickler = gl_pickle.GLPickler(filename = 'foo-bar')
    pickler.dump(obj)

    # The pickler has to be closed to make sure the files get closed.
    pickler.close()

User 1912 | 5/11/2015, 11:20:37 AM

how to load back the pickled object?


User 91 | 5/11/2015, 2:27:08 PM

unpickler = gl_pickle.GLUnPickler obj = unpickler.load()


User 2488 | 11/4/2015, 6:54:36 PM

When I try to do something similar to you @srikris , I get the following error message:

File "recommendationsscript2.py", line 28 in <module> from graphlab.util import glpickle ImportError: cannot import name glpickle

Any ideas?


User 5191 | 5/20/2016, 9:58:00 PM

@EvanSamanas @chusteven I get the same error message. Would love to be able to store models from grid searches.


User 5191 | 5/20/2016, 11:01:52 PM

@chusteven Just figured it out. The Graphlab pickler is nested within the main Graphlab module: gl._gl_pickle.GLPickler()


User 5230 | 5/31/2016, 3:10:30 PM

f = './itemitemrecogl.cppkl' print(f) unpickler = gl.gl_pickle.GLPickler(f) obj = unpickler.load() pickler.close()

obj = unpickler.load() AttributeError: 'GLPickler' object has no attribute 'load'

getting the above error. please guide what has to be done.


User 2014 | 5/31/2016, 10:51:20 PM

From @srikris, the load method exists in the GLUnPickler object, no GLPickler.


User 5230 | 6/1/2016, 6:52:49 AM

could you please give me a complete example for GLUnPickler


User 5230 | 6/1/2016, 9:17:53 AM

An example

pickler = gl.glpickle.GLPickler(filename = 'model_gl.cppkl') pickler.dump(m) pickler.close()

unpickle = gl.glpickle.GLUnpickler(filename='model_gl.cppkl')

    model_user_ratings = unpickle.load()
    unpickle.close()

User 2014 | 6/1/2016, 9:06:41 PM

This is an example: pickler = gl_pickle.GLPickler(filename) pickler.dump(obj) pickler.close() unpickler = gl_pickle.GLUnpickler(filename) obj_ret = unpickler.load() unpickler.close() # obj and obj_ret should be identical :smile: