Bug in Recommender Evaluation

User 1137 | 12/31/2014, 2:53:25 AM

I was trying to run the script from this post http://graphlab.com/learn/gallery/notebooks/recsysrank10K_song.html using GraphLab Create 1.2. The evaluation metrics were very low (pasted below) and do not match with those in the original post. However, when I switch back to GraphLab Create 1.1. The results are just fine. I suspect there might be a bug in the recommender.evaluate(dataset) method.

[WARNING] Model trained without a target. Skipping RMSE computation. compare_models: using 6868 users to estimate model performance PROGRESS: Evaluate model M0 PROGRESS: recommendations finished on 1000/6868 queries. users per second: 32370.8 PROGRESS: recommendations finished on 2000/6868 queries. users per second: 39172.7 PROGRESS: recommendations finished on 3000/6868 queries. users per second: 40872.5 PROGRESS: recommendations finished on 4000/6868 queries. users per second: 43104.8 PROGRESS: recommendations finished on 5000/6868 queries. users per second: 44435.4 PROGRESS: recommendations finished on 6000/6868 queries. users per second: 43456.8

Precision and recall summary statistics by cutoff +--------+------------------+------------------+ | cutoff | meanprecision | meanrecall | +--------+------------------+------------------+ | 2 | 0.00640652300524 | 0.00199693628735 | | 4 | 0.00425888177053 | 0.00271782557703 | | 6 | 0.00351873422636 | 0.0034035776156 | | 8 | 0.00291205591147 | 0.00372064913024 | | 10 | 0.00257716948165 | 0.0041538050416 | | 12 | 0.00229324403029 | 0.00460022270373 | | 14 | 0.00213204093519 | 0.00506130720704 | | 16 | 0.00195653756552 | 0.00534782366029 | | 18 | 0.00187665825406 | 0.00566351756916 | | 20 | 0.001776354106 | 0.00612666423936 | +--------+------------------+------------------+ [10 rows x 3 columns]

[WARNING] Model trained without a target. Skipping RMSE computation. PROGRESS: Evaluate model M1 PROGRESS: recommendations finished on 1000/6868 queries. users per second: 500.585 PROGRESS: recommendations finished on 2000/6868 queries. users per second: 508.75 PROGRESS: recommendations finished on 3000/6868 queries. users per second: 515.235 PROGRESS: recommendations finished on 4000/6868 queries. users per second: 515.482 PROGRESS: recommendations finished on 5000/6868 queries. users per second: 515.759 PROGRESS: recommendations finished on 6000/6868 queries. users per second: 516.293

Precision and recall summary statistics by cutoff +--------+------------------+------------------+ | cutoff | meanprecision | meanrecall | +--------+------------------+------------------+ | 2 | 0.0224956319161 | 0.00810898597421 | | 4 | 0.0139050669773 | 0.00996280361368 | | 6 | 0.0100708600272 | 0.0108600761462 | | 8 | 0.00791715200932 | 0.0115033325126 | | 10 | 0.00650844496214 | 0.0120607543262 | | 12 | 0.0055450397981 | 0.0122942365868 | | 14 | 0.0048256926533 | 0.0124512061567 | | 16 | 0.00429528246942 | 0.0126344293237 | | 18 | 0.0038746521711 | 0.0127988942996 | | 20 | 0.0035163075131 | 0.0130284824758 | +--------+------------------+------------------+ [10 rows x 3 columns]

Comments

User 1026 | 12/31/2014, 4:01:03 PM

Hi I have posted that problem already. They reproduced the bug and are about to fix it :-)


User 1137 | 12/31/2014, 5:29:47 PM

Hi blattnerma,

Glad to hear that! Hope they will fix it soon. :)


User 19 | 12/31/2014, 11:43:33 PM

Hi all,

Thank you both for posting. We have addressed the issue and uploaded a new version (1.2.1) to PyPI. You can upgrade with

<pre>pip install --upgrade graphlab-create </pre>

Please try it and let us know if you come across any issues.

Cheers, Chris


User 1137 | 1/2/2015, 6:46:13 PM

Hi Chris,

Thanks for the quick fix! It works well now.

Cheers, Chun


User 19 | 1/2/2015, 6:53:11 PM

Great!

Please do not hesitate to post any questions that arise. We are here to help.

Cheers, Chris