spark python code => ipython

User 5193 | 5/17/2016, 7:52:58 AM

Is it possible to convert the information to develop a spark python ipython? Code is:

from pyspark import SparkContext from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating sc = SparkContext()

Load and parse the data

data = sc.textFile("ui.txt") ratings = data.map(lambda l: l.split(','))\ .map(lambda l: Rating(int(l[0]), int(l[1]), int(l[2]))) ratings.cache()

Build the recommendation model using Alternating Least Squares

rank = 200 numIterations = 10 model = ALS.trainImplicit(ratings, rank, numIterations, 0.01, alpha=0.01, nonnegative=True)

Evaluate the model on training data

testdata = ratings.map(lambda p: (p[0], p[1])) predictions = model.predictAll(testdata).map(lambda r: ((r[0], r[1]), r[2])) ratesAndPreds = ratings.map(lambda r: ((r[0], r[1]), r[2])).join(predictions) MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).mean() print("Mean Squared Error = " + str(MSE))

Save model

model.save(sc, "./rec_model")

Comments

User 1774 | 5/17/2016, 10:04:20 AM

Hi @parkjoonho , I tried to make the conversion. Attached is the code. I did in in IPython Notebook (attached), and also converted the notebook to html and py files in case you don't have IPython Notebook. Let me know if that works for you.

Best,

g.


User 5193 | 5/18/2016, 6:32:04 AM

Hi@guyrap. Answers are very grateful to give. And one more question.


import sys from pyspark import SparkContext from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating

sc = SparkContext()

model = MatrixFactorizationModel.load(sc, "./rec_model") ratings = model.recommendProducts(int(sys.argv[1]), int(sys.argv[2]));

for r in ratings: print(str(r[1]))


$./spark-submit get_rec.list.py 4523039 5 (4523039 = user / 5= recommend item )