How to reduce predictions latency?

User 657 | 9/1/2014, 6:45:42 AM

Thank you for your great work, I really enjoy using Graphlab. I have some questions regarding GL's predictions latency. My use case for using GL would be in the advertising real time bidding space where response needs to be under 50ms. My goal is to keep predictions under 15ms. Currently the data to be predicted have 26 columns and 1 row.

Building and training model in GL is fast but I have some concerns with the predictions latency. Below are some numbers I've recorded.

The predict function itself is around 5-15ms which is ok <pre class="CodeBlock"><code>%timeit model.predict(data) 100 loops, best of 3: 11.4 ms per loop</code></pre>

Creating SFrame however, is much slower taking at least 50ms everytime I've tried. And since func:model.predict only takes in SFrame, this is a problem. <pre class="CodeBlock"><code>%timeit gl.SFrame(data) 10 loops, best of 3: 66 ms per loop</code></pre>

I've tried other methods like creating SFrame beforehand and modify each column's data directly but it still takes at least 10ms to modify each column's data.

Thanks

Comments

User 6 | 9/1/2014, 8:47:21 AM

Hi, We are currently working on a RESTFul API that will allow serving your online predictions, without having the load the model for each prediction. We would love working with you for trying it out once it is ready. Please send me a personal email and I will connect you with the relevant people in our team.


User 20 | 9/3/2014, 1:09:45 AM

Hi,

Thanks for running this benchmark! This is very useful. SFrame creation latency does seem ... quite high ... at the moment. We will definitely look into it optimizing that. It certainly should be much lower.

Thanks, Yucheng