数据科学包——Day2
利用Pandas, Numpy进行电影数据分析
准备工作
从网站grouplens.org/datasets/movielens 下载 MovieLens 1M Dataset 数据及数据说明书
任务：
- 数据读取
- 数据合并
- 统计电影平均得分
- 统计活跃电影——被评价的次数越多说明越活跃
- 女生最喜欢的电影
- 男生最喜欢的电影
- 男女生评分差异最大的电影——某类电影女生喜欢而男生不喜欢
- 最具争议的电影排行榜——评分的方差最大
import pandas as pd
import numpy as np
数据读取
'''
数据读取
'''
user_names=['user_id','gender','age','occupation','zip']
users=pd.read_table('ml-1m/users.dat', sep='::', header=None, names=user_names, engine='python')
rating_names = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings = pd.read_table('ml-1m/ratings.dat', sep='::', header=None, names=rating_names, engine='python')
movie_names = ['movie_id', 'title', 'genres']
movies = pd.read_table('ml-1m/movies.dat', sep='::', header=None, names=movie_names, engine='python')
print(len(users)
users.head()
'''
6040
user_id gender age occupation zip
0 1 F 1 10 48067
1 2 M 56 16 70072
2 3 M 25 15 55117
3 4 M 45 7 02460
4 5 M 25 20 55455
'''
print(len(ratings)
ratings.head()
'''
1000209
user_id movie_id rating timestamp
0 1 1193 5 978300760
1 1 661 3 978302109
2 1 914 3 978301968
3 1 3408 4 978300275
4 1 2355 5 978824291
'''
print(len(movies)
movies.head()
'''
3883
movie_id title genres
0 1 Toy Story (1995) Animation|Children's|Comedy
1 2 Jumanji (1995) Adventure|Children's|Fantasy
2 3 Grumpier Old Men (1995) Comedy|Romance
3 4 Waiting to Exhale (1995) Comedy|Drama
4 5 Father of the Bride Part II (1995) Comedy
'''
数据合并
data=pd.merge(pd.merge(users,ratings),movies)
data.head(5)
'''
user_id gender age occupation zip movie_id rating timestamp title genres
0 1 F 1 10 48067 1193 5 978300760 One Flew Over the Cuckoo's Nest (1975) Drama
1 2 M 56 16 70072 1193 5 978298413 One Flew Over the Cuckoo's Nest (1975) Drama
2 12 M 25 12 32793 1193 4 978220179 One Flew Over the Cuckoo's Nest (1975) Drama
3 15 M 25 7 22903 1193 4 978199279 One Flew Over the Cuckoo's Nest (1975) Drama
4 17 M 50 1 95350 1193 5 978158471 One Flew Over the Cuckoo's Nest (1975) Drama
'''
按性别查看各个电影的平均评分
mean_ratings_gender=data.pivot_table(values='rating',index='title',columns='gender',aggfunc='mean'
男女评分差异最大的电影
mean_ratings_gender['diff']=mean_ratings_gender.F-mean_ratings_gender.M
mean_ratings_gender.head(5)
'''
gender F M diff
title
$1,000,000 Duck (1971) 3.375000 2.761905 0.613095
'Night Mother (1986) 3.388889 3.352941 0.035948
'Til There Was You (1997) 2.675676 2.733333 -0.057658
'burbs, The (1989) 2.793478 2.962085 -0.168607
...And Justice for All (1979) 3.828571 3.689024 0.139547
'''
'''
按差值排序
'''
mean_ratings_gender.sort_values(by='diff',ascending=True)
活跃电影排行
ratings_by_movie_title=data.groupby('title').size()
ratings_by_movie_title.head(5)
'''
title
$1,000,000 Duck (1971) 37
'Night Mother (1986) 70
'Til There Was You (1997) 52
'burbs, The (1989) 303
...And Justice for All (1979) 199
dtype: int64
'''
前十大活跃电影——被评价次数超过1000的电影top10
top_ratings=ratings_by_movie_title[ratings_by_movie_title>1000]
top_10_ratings=top_ratings.sort_values(ascending=False).head(10)
'''
title
American Beauty (1999) 3428
Star Wars: Episode IV - A New Hope (1977) 2991
Star Wars: Episode V - The Empire Strikes Back (1980) 2990
Star Wars: Episode VI - Return of the Jedi (1983) 2883
Jurassic Park (1993) 2672
Saving Private Ryan (1998) 2653
Terminator 2: Judgment Day (1991) 2649
Matrix, The (1999) 2590
Back to the Future (1985) 2583
Silence of the Lambs, The (1991) 2578
dtype: int64
'''
前二十大高分电影——评分分最高的电影
mean_ratings=data.pivot_table(values='ratings',index='title',aggfunc='mean')
top_20_mean_ratings=mean_ratings.sort_values(ascending=False).head(20)
'''
title
Gate of Heavenly Peace, The (1995) 5.000000
Lured (1947) 5.000000
Ulysses (Ulisse) (1954) 5.000000
Smashing Time (1967) 5.000000
Follow the Bitch (1998) 5.000000
Song of Freedom (1936) 5.000000
Bittersweet Motel (2000) 5.000000
Baby, The (1973) 5.000000
One Little Indian (1973) 5.000000
Schlafes Bruder (Brother of Sleep) (1995) 5.000000
I Am Cuba (Soy Cuba/Ya Kuba) (1964) 4.800000
Lamerica (1994) 4.750000
Apple, The (Sib) (1998) 4.666667
Sanjuro (1962) 4.608696
Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954) 4.560510
Shawshank Redemption, The (1994) 4.554558
Godfather, The (1972) 4.524966
Close Shave, A (1995) 4.520548
Usual Suspects, The (1995) 4.517106
Schindler's List (1993) 4.510417
Name: rating, dtype: float64
'''
前十大活跃电影平均分——不一定越活跃的电影评分越高
mean_ratings[top_10_ratings.index]
'''
title
American Beauty (1999) 4.317386
Star Wars: Episode IV - A New Hope (1977) 4.453694
Star Wars: Episode V - The Empire Strikes Back (1980) 4.292977
Star Wars: Episode VI - Return of the Jedi (1983) 4.022893
Jurassic Park (1993) 3.763847
Saving Private Ryan (1998) 4.337354
Terminator 2: Judgment Day (1991) 4.058513
Matrix, The (1999) 4.315830
Back to the Future (1985) 3.990321
Silence of the Lambs, The (1991) 4.351823
Name: rating, dtype: float64
'''
前二十大高分电影的活跃程度——不一定越高分的电影越活跃，可能评价的人少但是均分很高
ratings_by_movie_title[top_20_mean_ratings.index]
'''
title
Gate of Heavenly Peace, The (1995) 3
Lured (1947) 1
Ulysses (Ulisse) (1954) 1
Smashing Time (1967) 2
Follow the Bitch (1998) 1
Song of Freedom (1936) 1
Bittersweet Motel (2000) 1
Baby, The (1973) 1
One Little Indian (1973) 1
Schlafes Bruder (Brother of Sleep) (1995) 1
I Am Cuba (Soy Cuba/Ya Kuba) (1964) 5
Lamerica (1994) 8
Apple, The (Sib) (1998) 9
Sanjuro (1962) 69
Seven Samurai (The Magnificent Seven) (Shichinin no samurai) (1954) 628
Shawshank Redemption, The (1994) 2227
Godfather, The (1972) 2223
Close Shave, A (1995) 657
Usual Suspects, The (1995) 1783
Schindler's List (1993) 2304
dtype: int64
'''
十大好电影——活跃度高，均分也高
top_10_movies=mean_ratings[top_ratings.index].sort_values(ascending=False).head(10)
'''
title
Shawshank Redemption, The (1994) 4.554558
Godfather, The (1972) 4.524966
Usual Suspects, The (1995) 4.517106
Schindler's List (1993) 4.510417
Raiders of the Lost Ark (1981) 4.477725
Rear Window (1954) 4.476190
Star Wars: Episode IV - A New Hope (1977) 4.453694
Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963) 4.449890
Casablanca (1942) 4.412822
Sixth Sense, The (1999) 4.406263
Name: rating, dtype: float64
'''
综合均分与活跃度
df_top_10_movies=pd.DataFrame(top_10_movies)
df_top_10_movies['hot']=top_ratings[top_10_movies.index]
'''
rating hot
title
Shawshank Redemption, The (1994) 4.554558 2227
Godfather, The (1972) 4.524966 2223
Usual Suspects, The (1995) 4.517106 1783
Schindler's List (1993) 4.510417 2304
Raiders of the Lost Ark (1981) 4.477725 2514
Rear Window (1954) 4.476190 1050
Star Wars: Episode IV - A New Hope (1977) 4.453694 2991
Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963) 4.449890 1367
Casablanca (1942) 4.412822 1669
Sixth Sense, The (1999) 4.406263 2459
'''
完结！
948
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
