Loading an SGraph from a database?

User 960 | 11/19/2014, 12:51:00 PM

Hi, I'm interested in finding the most efficient way to load an SGraph from postgres. I'm considering 2 options (maybe there are others I'm unaware of). I can't assume the graph will fit into memory.

  • Option 1: The obvious option, export data from postgres to a supported file format

  • Option 2: Stream data into GraphLab using psycopg2 (or the like)

A few questions related to option 2:

  • Is this possible to update an SGraph iteratively (say 1M edges at a time)?

  • Assuming it's possible, is it efficient compared to file ingestion (Option 1) or some other option?

Comments

User 14 | 11/19/2014, 10:56:15 PM

Hi @SmartTypes‌

Right now the best choise will be to export the data from postgres into csv and read_csv into SFrame. You can then add SFrame into a new SGraph or an existing SGraph iteratively.

I have not used psycopg2 and cannot comment on its efficiency. But Option 1 is easier and less error prone. We are working on ODBC connectors. Please stay tuned.