Pyspark:电影推荐

目录

Data set

  来自一个开源电影数据集,约10万条数据。

Create SparkSession and Load data

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('RS').getOrCreate()

# load dataset
df = spark.read.csv('./Data/movie_ratings_df.csv', inferSchema=True, header=True)

EDA

df.show(10)
+------+------------+------+
|userId|       title|rating|
+------+------------+------+
|   196|Kolya (1996)|     3|
|    63|Kolya (1996)|     3|
|   226|Kolya (1996)|     5|
|   154|Kolya (1996)|     3|
|   306|Kolya (1996)|     5|
|   296|Kolya (1996)|     4|
|    34|Kolya (1996)|     5|
|   271|Kolya (1996)|     4|
|   201|Kolya (1996)|     4|
|   209|Kolya (1996)|     4|
+------+------------+------+
only showing top 10 rows
print((df.count(), len(df.columns)))
(100000, 3)
df.printSchema()
root
 |-- userId: integer (nullable = true)
 |-- title: string (nullable = true)
 |-- rating: integer (nullable = true)
# user recording
userCount_df = df.groupBy('userId').count().orderBy('count', ascending=False)

# Spark's DataFrame --> Pandas's DataFrame
user_df = userCount_df.toPandas()
user_df.head()
userIdcount
0405737
1655685
213636
3450540
4276518
user_df.tail()
userIdcount
93824220
93957120
94087320
94147520
9423620

Feature Engineering

  • label encode
    • Instance = stringIndexer(inputCol=‘xxx’, outputCol=‘yyy’)
    • model = Instance.fit(df)
    • new_df = model.transform(df)
  • reverse operation
    • Instance = IndexToString(inputCol=‘yyy’, outputCol=‘xxx’, labels=model.labels)
    • df = Instance.transform(new_df)
from pyspark.sql.functions import *
from pyspark.ml.feature import StringIndexer, IndexToString
stringIndexer = StringIndexer(inputCol='title', outputCol='title_num')

model = stringIndexer.fit(df)

new_df  = model.transform(df)

new_df.show(10)
+------+------------+------+---------+
|userId|       title|rating|title_num|
+------+------------+------+---------+
|   196|Kolya (1996)|     3|    287.0|
|    63|Kolya (1996)|     3|    287.0|
|   226|Kolya (1996)|     5|    287.0|
|   154|Kolya (1996)|     3|    287.0|
|   306|Kolya (1996)|     5|    287.0|
|   296|Kolya (1996)|     4|    287.0|
|    34|Kolya (1996)|     5|    287.0|
|   271|Kolya (1996)|     4|    287.0|
|   201|Kolya (1996)|     4|    287.0|
|   209|Kolya (1996)|     4|    287.0|
+------+------------+------+---------+
only showing top 10 rows
new_df.groupBy('title_num').count().orderBy('count', ascending=False).show(10)
+---------+-----+
|title_num|count|
+---------+-----+
|      0.0|  583|
|      1.0|  509|
|      2.0|  508|
|      3.0|  507|
|      4.0|  485|
|      5.0|  481|
|      6.0|  478|
|      7.0|  452|
|      8.0|  431|
|      9.0|  429|
+---------+-----+
only showing top 10 rows

Splitting Dataset

train_df, test_df = new_df.randomSplit([0.7, 0.3])

print('train_df, (%d, %d)'%(train_df.count(), len(train_df.columns)))
print('test_df, (%d, %d)'%(test_df.count(), len(test_df.columns)))
train_df, (69917, 4)
test_df, (30083, 4)

Train Model

ALS

  • nonnegative=‘True’, 没有分数为负值的评分
  • coldStartStrategy=‘drop’,防止冷启动,分数为NaN
from pyspark.ml.recommendation import ALS

rec = ALS(maxIter=10, regParam=0.01, userCol='userId', itemCol='title_num', ratingCol='rating', nonnegative=True,
                 coldStartStrategy='drop')

rs_model = rec.fit(train_df)

Evaluation

test_df.show(4)
+------+--------------------+------+---------+
|userId|               title|rating|title_num|
+------+--------------------+------+---------+
|     1|Apocalypse Now (1...|     3|     91.0|
|     1|    Apollo 13 (1995)|     4|     51.0|
|     1|Aristocats, The (...|     2|    575.0|
|     1|Army of Darkness ...|     4|    289.0|
+------+--------------------+------+---------+
only showing top 4 rows
test = test_df.select(['userId','title','title_num'])

test_pred = rs_model.transform(test)

test_pred.show()
+------+--------------------+---------+----------+
|userId|               title|title_num|prediction|
+------+--------------------+---------+----------+
|   463|That Thing You Do...|    148.0| 2.5579684|
|   251|That Thing You Do...|    148.0| 3.9390717|
|   193|That Thing You Do...|    148.0|  3.070894|
|   642|That Thing You Do...|    148.0| 3.9546824|
|   101|That Thing You Do...|    148.0| 3.6771653|
|   406|That Thing You Do...|    148.0| 3.0834832|
|   731|That Thing You Do...|    148.0| 2.8964741|
|   159|That Thing You Do...|    148.0|  5.029048|
|   606|That Thing You Do...|    148.0| 3.6251612|
|   336|That Thing You Do...|    148.0| 2.9238327|
|   330|That Thing You Do...|    148.0| 4.2745767|
|    93|That Thing You Do...|    148.0| 3.2026477|
|   654|That Thing You Do...|    148.0| 4.2684965|
|   152|That Thing You Do...|    148.0| 4.3452826|
|   760|That Thing You Do...|    148.0|  4.768305|
|   178|That Thing You Do...|    148.0| 4.1157365|
|   839|That Thing You Do...|    148.0| 4.2218776|
|   500|That Thing You Do...|    148.0| 2.9480288|
|   432|That Thing You Do...|    148.0| 2.7699072|
|   676|That Thing You Do...|    148.0| 2.6120367|
+------+--------------------+---------+----------+
only showing top 20 rows
from pyspark.ml.evaluation import RegressionEvaluator

test_pred = rs_model.transform(test_df)

evaluate_result = RegressionEvaluator(metricName='rmse', predictionCol='prediction', labelCol='rating')

rmse = evaluate_result.evaluate(test_pred)

print('test rmse is %f'%rmse)
test rmse is 1.036787

Recommend Movies

nunique_movies = new_df.select('title').distinct()

nunique_movies.count()
1664
# total movies
a = nunique_movies.alias('a')
user_id = 66
# user_id = 66,看过的电影
watched_movies = new_df.filter(new_df['userId'] == user_id).select('title').distinct()
b = watched_movies.alias('b')
# a join b
total_movies = a.join(b, a.title == b.title, how='left')
total_movies.show(25)
+--------------------+--------------------+
|               title|               title|
+--------------------+--------------------+
|   Annie Hall (1977)|                null|
|Heavenly Creature...|                null|
|       Psycho (1960)|                null|
|Snow White and th...|                null|
|Night of the Livi...|                null|
|When We Were King...|                null|
| If Lucy Fell (1996)|                null|
|    Fair Game (1995)|                null|
| Three Wishes (1995)|                null|
|         Cosi (1996)|                null|
|Paris, France (1993)|                null|
|Spanking the Monk...|                null|
|I'll Do Anything ...|                null|
|        Mondo (1996)|                null|
| Evil Dead II (1987)|                null|
|    Threesome (1994)|                null|
|Last Action Hero ...|                null|
|Reality Bites (1994)|                null|
|Colonel Chabert, ...|                null|
|   Blue Chips (1994)|                null|
|A Chef in Love (1...|                null|
|    Nico Icon (1995)|                null|
|English Patient, ...|English Patient, ...|
|Marvin's Room (1996)|                null|
|Crows and Sparrow...|                null|
+--------------------+--------------------+
only showing top 25 rows
user_66_not_watched_movies = total_movies.where(col('b.title').isNull()).select(a.title).distinct()

print('user 66 没有看过的电影 %d'%user_66_not_watched_movies.count())
user 66 没有看过的电影 1626
user_66_not_watched_movies = user_66_not_watched_movies.withColumn('userId', lit(int(user_id)))

user_66_not_watched_movies.show(10, False)
+--------------------------------------+------+
|title                                 |userId|
+--------------------------------------+------+
|Annie Hall (1977)                     |66    |
|Heavenly Creatures (1994)             |66    |
|Psycho (1960)                         |66    |
|Snow White and the Seven Dwarfs (1937)|66    |
|Night of the Living Dead (1968)       |66    |
|When We Were Kings (1996)             |66    |
|If Lucy Fell (1996)                   |66    |
|Fair Game (1995)                      |66    |
|Three Wishes (1995)                   |66    |
|Cosi (1996)                           |66    |
+--------------------------------------+------+
only showing top 10 rows

Top 10 Recommend Movies

user_66_df = model.transform(user_66_not_watched_movies)   # title >> title_num

user_66_rs = rs_model.transform(user_66_df).orderBy('prediction', ascending=False)

user_66_rs.show(10)
+--------------------+------+---------+----------+
|               title|userId|title_num|prediction|
+--------------------+------+---------+----------+
|Ruby in Paradise ...|    66|    887.0|  6.453057|
|Misrables, Les (...|    66|    911.0|  5.965375|
| Apostle, The (1997)|    66|    572.0|  5.873897|
|Miami Rhapsody (1...|    66|   1022.0| 5.8026986|
|  Schizopolis (1996)|    66|   1372.0|  5.692012|
|       Harlem (1993)|    66|   1369.0|  5.661992|
|Mina Tannenbaum (...|    66|   1286.0|  5.637907|
|In the Bleak Midw...|    66|    996.0|  5.628146|
|Double vie de V...|    66|    904.0|  5.467951|
|     Faithful (1996)|    66|   1129.0| 5.2826753|
+--------------------+------+---------+----------+
only showing top 10 rows
评论 4
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值