Accuracy discrepancy in BoostedTrees

User 1095 | 12/18/2014, 1:38:40 PM

Hello!

We find an issue in the evaluation of a prediction using Boosted Trees. When evaluating using the evaluate method of the own classifier we get an accuracy of "0.9864864864864865", which is false, when using the evaluation module, like "graphlab.evaluation.accuracy(pred, test)" we get an accuracy of 0.5933660933660934. The confusion matrix seems to be the same using both methos.

Using a the Neural Network as a classifier we have a similar issue, the accuracy given is 0.03500000014901161 and the evaluation.accuracy returns 0.03814713896457766, a small but still a difference.

Hope to hear from you soon

Regards

Comments

User 91 | 12/18/2014, 10:58:00 PM

Thanks for catching these issues. We hope enjoy using GraphLab create.

The Neural Network as a classifier uses single precision for computing accuracy while the evaluation.accuracy uses double precision. This explains the reasons for the difference. We use single precision for Neural networks because GPU memory is very precious, and single precision is 2x more efficient.

With respect to boosted trees classifier, it could be an issue that we would like to look into. Can you share your dataset with us (if it is public). If not, feel free to email me at srikris@graphlab.com and we do this privately.


User 91 | 12/19/2014, 2:58:59 AM

We were able to reproduce the issue. It seems that the boosted trees classifier's evaluation had an issue in the multi-class classification scenario.

The issue has been isolated and the fix will be in during the next release (which is coming very soon). Thanks for your patience.