pyspark学习——最流行的电影

数据集下载：https://grouplens.org/datasets/movielens/

选择 older datasets下载

在readme.txt中可以看到u.data的数据格式：

u.data     -- The full u data set, 100000 ratings by 943 users on 1682 items.
              Each user has rated at least 20 movies.  Users and items are
              numbered consecutively from 1.  The data is randomly
              ordered. This is a tab separated list of 
	         user id | item id | rating | timestamp. 
              The time stamps are unix seconds since 1/1/1970 UTC   

 

import findspark
findspark.init()
from pyspark import SparkConf,SparkContext
conf = SparkConf().setMaster('local').setAppName('Movies')
sc = SparkContext(conf = conf)

def get_movie_names():
    movieNames = {}
    with open('u.item',encoding='iso-8859-15') as f:
        for line in f:
            fields =line.split("|")
            movieNames[int(fields[0])] = fields[1]
    return movieNames
movieDict =  sc.broadcast(get_movie_names())

lines =sc.textFile("u.data")
movies = lines.map(lambda x :(int(x.split()[1]),1))
moviesCounts = movies.reduceByKey(lambda x,y : x + y)
#但是我们希望按照评分人数进行排序，因此我们需要交换一下
flipped = moviesCounts.map(lambda x: (x[1],x[0]))
sortedMovies = flipped.sortByKey(ascending = False)#降序排序
# results = sortedMovies.collect()
# for result in results:
#     print(result)
    
#不知道电影的名字，因此我们需要解决这个问题。电影名字在u.item这个文件中。
#broadcast的方式来计算，效率高  电影id和电影名字的表

#现在我们只需要将排序好的电影id映射为电影名字即可
sortedMovieswithNames = sortedMovies.map(lambda x:(movieDict.value[x[1]],x[0]))
results = sortedMovieswithNames.collect()
for result in results:
    print(result)
'''
部分结果
('Star Wars (1977)', 583)
('Contact (1997)', 509)
('Fargo (1996)', 508)
('Return of the Jedi (1983)', 507)
('Liar Liar (1997)', 485)
('English Patient, The (1996)', 481)
('Scream (1996)', 478)
'''