Strange recall and precision on version 1.2

User 1026 | 12/27/2014, 3:00:42 PM

Hi there

I updated Graphlab Create to the new version 1.2 (from version 1.1). I run the item similarity recommender algorithm on exactly the same dataset with the same parameters and got completely different results in version 1.2 and version 1.1. In version 1.2 I got zeros for recall and precision, whereas I got 0.15 for recall and 0.03 for precision in version 1.1. What did change? I could not find anything in the documentation.

Comments

User 19 | 12/27/2014, 5:52:14 PM

Hi,

Thanks for getting in touch. Can you provide some of the code that you are running? A small, reproducible example will help us track things down.

Chris


User 1026 | 12/27/2014, 10:09:03 PM

I run the trainmodel from the following code. It basically does a cross validation for every hyper parameter combination on a grid (constructed from the provided parameters). Using the simple itemsimilarity model gives completely different results in version 1.1 and 1.2 on recall and precision (sometimes a factor 150). Even if the splits in train and test set differ and therefore we can expect different results I hardly can believe that they can differ in that way.

import graphlab as gl import itertools import numpy as np

MODELS = ['popularity','similarity','ranking_factorization','factorization']

def modeltrain(sf,model,defaultpar,hyperpar,numinstance,resultpath,maxnumusers=1000,splitratio=0.2): """

Trains a method (algorithm)

Parameters:
-----------
sf: SFrame 
model: graphlab model object (algorithm)
default_par: default parameter (user_id,item_id,target_id [optional]) (dict)
hyper_par: hyperparamter of the corresponding model (list of dict), e.g., [{'num_factor':[10,10,30],'regularization':[.0,0.1]}]
split_ratio: train, test ratio
max_num_users: max number of users for split  
num_instance: number of instances (train test splits)
result_path: path of the SFrame containing the results (this is an absolut path)

Example:
--------
sfperf=rec.model_train(sf,'similarity',{'user_id':'userId','item_id':'trackCode'},[{'only_top_k':[10,20]}],2,'/Users/blattner/')


Output:
-------
SFrame 	
"""
if model not in MODELS:
	print "Model: %s not known" %model  
	return
elif model == 'popularity':
	mod = gl.recommender.popularity_recommender.create
	mod_inst = gl.recommender.popularity_recommender
elif model == 'similarity':
	mod = gl.recommender.item_similarity_recommender.create
	mod_inst = gl.recommender.item_similarity_recommender
elif model == 'ranking_factorization':
	mod = gl.recommender.ranking_factorization_recommender.create
	mod_inst = gl.recommender.ranking_factorization_recommender
elif model == 'factorization':
	mod = gl.recommender.factorization_recommender.create
	mod_inst = gl.recommender.factorization_recommender


#get default parameter of model
l=list(mod_inst.get_default_options()['name'])

#check if keywords in default_par exist for selected model
k=default_par.keys()
false_par = [k[x] for x in range(len(k)) if k[x] not in l]
if len(false_par) > 0:
	print "Parameter %s not default parameter of %s model" % (false_par,model)
	return

#generate keys and grid for hyperparameter search
keys,grid=generate_parameter_grid(hyper_par)

#check if keywords in hyper_par exist for selected model
false_par = [keys[x] for x in range(len(keys)) if keys[x] not in l]
print false_par
if len(false_par) > 0:
	print "Parameter %s not hyper parameter of %s model" % (false_par,model)
	return


#assemble all configurations from the passed hyper_par and append them in a list (conf)
conf = list()
for i in grid:
	conf.append(dict(zip(keys,i)))

#train model with cross-validation for each parameter configuration
z=1
for c in conf:
	default_par.update(c)
	rmse_perf = list()
	prec_perf = list()
	rec_perf = list()
	for num in range(num_instance):
		#generate split for training and test set
		print " "
		print "********************************"
		print "Start training with parameter %s, instance number %s" % (default_par,str(num+1)) 
		print "********************************"
		print " "
		print "Split data set with %s user and ratio %s" % (str(max_num_users),str(split_ratio))
		print " "
		train,test = gl.recommender.util.random_split_by_user(sf,user_id=default_par['user_id'],item_id=default_par['item_id'],max_num_users=max_num_users,item_test_proportion=split_ratio)
		mod_train = mod(train,**default_par)
		print " "
		print "Evalulate precision-recall"
		print " "
		eval_pre_rec = mod_train.evaluate_precision_recall(test)
		prec_perf.append(list(eval_pre_rec['precision_recall_overall']['precision']))
		rec_perf.append(list(eval_pre_rec['precision_reHTTP/1.1 200 OK

Transfer-Encoding: chunked Date: Thu, 21 Jul 2016 23:13:36 GMT Server: Warp/3.2.6 Content-Type: application/json

016A ["37zyefqi2sweveyp","42fn7zeo6v5ui427","66pt5sk2wz2jrbzu","awoljknjigytdyls","cj2lanoogknwopto","cnm3adnh35xmsx3f","ebxs4t2y6xr5izzy","eg5zus2pz72mr7xb","exshwew2w2jv3n7r","hxrxgzvgms3incmf","hymu5oh2f5ctk5jr","jkisbjnul226jria","lag7djeljbjng6bu","o3l65o4qzcxs327j","qsk2jzo2zh523r24","t7k6g7fkndoggutd","xfllvjyax4inadxh","ygtjzi2wkfonj3z7","yycjajwpguyno4je"] 0


User 19 | 12/28/2014, 1:34:55 AM

Thanks, this is very useful. We will be looking into it.


User 1026 | 12/30/2014, 8:07:43 AM

Hi guys Any news about the recall-precision stuff?


User 19 | 12/30/2014, 5:24:33 PM

We have reproduced the issue and are currently looking into a fix. Thanks for your patience.


User 1026 | 12/30/2014, 6:17:10 PM

Thx. Sure


User 19 | 12/31/2014, 11:44:43 PM

Hi,

We have addressed the issue and uploaded a new version (1.2.1) to PyPI. You can upgrade with

<pre>pip install --upgrade graphlab-create </pre>

Please try it and let us know if you come across any issues.

Cheers, Chris


User 1026 | 1/1/2015, 9:50:41 AM

Hi Chris

Thanks for the warp-speed fix. It works! Happy New Year. Cheers Marcel