What is the implementation detail behind the GraphLab Create C++/Python Interop ?

User 2098 | 8/17/2015, 8:09:36 AM

I am not sure if this question should be asked in the forum, but I am so curious to know the answer.

Thanks to the effort of Dato Team, we can now write the extension and expose the extension to Python soooooo easily.

I was so surprised that the tool can be made in such an easy way, and very curious how this was implemented.

After some exploration, I get some pieces of information:

  1. To make an function extension, users need to write codes such as:


These Macros in essence is to create a function named gettoolkitfunction_registration() and it returns a vector of function specifications. This function is to be called at runtim by GraphLab. But how does GraphLab Create let python know that there is a new function ?

  1. Some code in the Dato-Core tells me that it uses Cython to make Python know the exist of the new C++ extension, but it seems GraphLab Create uses a different method to make C++ extension visible to Python. So again how does GraphLab Create achieve the python/c++ interop?

Please forgive my poor english, I am looking forward to hearing from your advice, just a direct link to the related code would be much appreciated.

Thanks in advance.


User 1189 | 8/17/2015, 6:18:41 PM

Details are complicated as always, but the short summary is as follow:

The interop uses a collection of C++ template magic to decompose the type (you can see it in https://github.com/dato-code/GraphLab-Create-SDK/tree/master/graphlab/unity/lib , the toolkitfunction* and toolkitclass* headers) into an internal object called the flexibletype or the varianttype.

The python side then interprets this (look at extensions.py) and generates the appropriate python functions and classes at runtime.

User 1189 | 8/17/2015, 6:18:59 PM

We are open sourcing this bit, so look out for it in the next 3 weeks or so.