TypeError: model_parameter_search() got multiple values for keyword argument 'standard_m

User 875 | 4/20/2015, 7:26:22 PM

Hi all,

I upgraded my Graphlab version from 1.0.1 to 1.1. Then I ran my code and wanted to execute the method modelparametersearch(). I got next TypeError:

<pre class="CodeBlock"><code>graphlab.toolkits.model_parameter_search(env, graphlab.factorization_recommender.create, train_file_path, '/home/user/LWS/team-codes/Projects/lws/PRE_CTI/data/results_2.gl', test_file_path, standard_model_params = {'target': 'Target','user_id':'userId','item_id':'itemId'}, hyper_params={'num_factors':[1,2,3]})

TypeError Traceback (most recent call last) <ipython-input-14-491f55bf0b85> in <module>() ----> 1 graphlab.toolkits.modelparametersearch(env, graphlab.factorizationrecommender.create, trainfilepath, '/home/user/LWS/team-codes/Projects/lws/PRECTI/data/results2.gl', testfilepath, standardmodelparams = {'target': 'Target','userid':'userId','itemid':'itemId'}, hyperparams={'num_factors':[1,2,3]})

TypeError: modelparametersearch() got multiple values for keyword argument 'standardmodelparams'</code></pre>

I didn't get it before with Graphlab 1.0.1 for the same input

Comments

User 4 | 4/21/2015, 7:29:19 AM

@Guforu: the API changed between 1.0 and 1.1, in order to more closely align with other similar APIs (like <code>graphlab.deploy.job.create</code>). Try this syntax instead (note that env is now a keyword argument rather than a positional argument): <pre><code>graphlab.toolkits.modelparametersearch(graphlab.factorizationrecommender.create, trainfilepath, '/home/user/LWS/team-codes/Projects/lws/PRECTI/data/results2.gl', testfilepath, standardmodelparams = {'target': 'Target','userid':'userId','itemid':'itemId'}, hyperparams={'num_factors':[1,2,3]}, environment=env) </code></pre>


User 875 | 4/21/2015, 7:40:21 AM

Dear Zach, we try to ugrade every time to the new version of Graphlab a squickly as possible. Now we use already version 1.3.0 and I can evaluate the modelparametersearch method regarding the new documantation on: https://dato.com/products/create/docs/graphlab.toolkits.modelparametersearch.html I have just one small question. If I do grid search for my recommender, is it possible to change recommender list length from 5 (default) to other values. With other words change "trainingprecision@5" and "trainingrecall@5" to "trainingprecision@10" or "traininngprecision@12" respectively the same for recall?


User 4 | 4/21/2015, 7:58:40 AM

You should be able to change the evaluation metric by setting a custom evaluator function with <code>evaluator=</code> as documented <a href="https://dato.com/products/create/docs/graphlab.toolkits.modelparametersearch.html">here</a>. Take a look at the definition of a custom evaluator function there (you will need to make one specific to recommender, but it can work as you desire).


User 875 | 4/21/2015, 8:13:40 AM

ok, thank you a lot


User 875 | 4/21/2015, 1:10:01 PM

Dear Zach, I analyzed your solution with <code class="CodeInline">evaluator=</code> and I don't think this is a solution on my question. I will describe my problem more precisely. For example we create a model for our recommender (all hyperparameters are given and stay fix). Now, we want to evaluate the precision and recall for our model. The definition of precision is given by TP/(TP+FP). We want to evaluate this value for the different lengths of our recommendation list for the same model. Actually we want to see, how the precision is changed, if we recommend 20 items instead 5. I can define the length for my recommendation list in the <code class="CodeInline">recommend()</code> function:

<pre class="CodeBlock"><code>model = graphlab.rankingfactorizationrecommender.create(....) rec = model.recommend(k=10)</code></pre>

Now, I didn't find this parameter in the <code class="CodeInline">def customevaluator(model, train, test)</code>. How we understood early this parameter is definied only in the <code class="CodeInline">recommend()</code> method, not in hyper parameters. In the <code class="CodeInline">modelparameter_search()</code> the output for precision has the fix length:

<pre class="CodeBlock"><code> +----------+-------------+----------------------+-------------------+ | modelid | numfactors | trainingprecision@5 | trainingrecall@5 | +----------+-------------+----------------------+-------------------+ | 0 | 1 | 0.1 | 0.5 | | 1 | 5 | 0.1 | 0.5 | +----------+-------------+----------------------+-------------------+ </code></pre>

We want to have different lengths, for example:

<pre class="CodeBlock"><code> +----------+-------------+----------------------+-------------------+ | modelid | numfactors | trainingprecision@10| trainingrecall@10 | +----------+-------------+----------------------+-------------------+ | 0 | 1 | 0.08 | 0.45 | | 1 | 5 | 0.07 | 0.52 | +----------+-------------+----------------------+-------------------+ </code></pre>

Is it possible to do this with the <code class="CodeInline">modelparametersearch()</code> method?


User 875 | 4/21/2015, 1:14:50 PM

Some parameter, like <code class="CodeInline">cutoff</code> here: https://dato.com/products/create/docs/generated/graphlab.recommender.itemsimilarityrecommender.ItemSimilarityRecommender.evaluateprecisionrecall.html


User 4 | 4/21/2015, 5:50:54 PM

Hi @Guforu, thanks for the explanation of what you are doing. I think <code>modelparametersearch</code> is too specific for this purpose (it is really meant for searching the hyperparameter space) and to do what you want, you only need to use the <code>evaluateprecisionrecall</code> like you are linking to. You can vary the <code>cutoff</code> parameter (or provide a list of multiple cutoff values) to get the desired effect.

If you want to perform a hyperparameter search in combination with evaluating precision/recall at multiple cutoffs, you can create a custom evaluator function for the modelparametersearch, and in that custom evaluator function, use m.evaluateprecisionrecall.


User 875 | 4/22/2015, 7:21:33 AM

ok, thank you