PySpark简单使用(五)——综合案例1

案例介绍

有如下的订单文件 orders.txt

{"id":1,"timestamp":"2019-05-08T01:03.00Z","category":"平板电脑","areaName":"北京","money":"1450"}|{"id":2,"timestamp":"2019-05-08T01:01.00Z","category":"手机","areaName":"北京","money":"1450"}|{"id":3,"timestamp":"2019-05-08T01:03.00Z","category":"手机","areaName":"北京","money":"8412"}
{"id":4,"timestamp":"2019-05-08T05:01.00Z","category":"电脑","areaName":"上海","money":"1513"}|{"id":5,"timestamp":"2019-05-08T01:03.00Z","category":"家电","areaName":"北京","money":"1550"}|{"id":6,"timestamp":"2019-05-08T01:01.00Z","category":"电脑","areaName":"杭州","money":"1550"}

需求是：

(1)获取城市销售额排名

(2) 全部城市有哪些商品类别在售卖

(3)北京市有哪些商品在售卖

from pyspark import SparkConf, SparkContext
import json
import os
# 1.让PySpark知道Python的解释器位置
os.environ['PYSPARK_PYTHON'] = "C:/Python310/python.exe"
conf = SparkConf().setMaster("local[*]").setAppName("test_spark")
sc = SparkContext(conf=conf)

# 需求1：获取城市销售额排名
# 1.1 读取文件得到RDD
file_rdd = sc.textFile("./orders.txt")
# 1.2 取出每个JSON串
json_str_rdd = file_rdd.flatMap(lambda x: x.split("|"))

# 1.3将每个json串转换为字典
dict_rdd = json_str_rdd.map(lambda x: json.loads(x))

# 1.4 取出城市和销售额数据组成元组
city_with_money_rdd = dict_rdd.map(lambda x: (x['areaName'], int(x['money'])))

# 1.5 按城市分组按销售额聚合
city_result_rdd = city_with_money_rdd.reduceByKey(lambda a, b: a + b)

# 1.6 按销售额聚合结果进行排序
result1_add = city_result_rdd.sortBy(lambda x: x[1], ascending=False, numPartitions=1)
print("城市销售额排名为：",result1_add.collect())

# 需求2：全部城市有哪些商品类别在售卖
# 2.1 取出全部的商品类别
category_rdd = dict_rdd.map(lambda x: x['category']).distinct()
print("全部城市售卖的商品类型：", category_rdd.collect())

# 需求3：北京市有哪些商品在售卖
# 3.1 过滤北京市数据
bj_data_rdd = dict_rdd.filter(lambda x: x['areaName'] == '北京')

# 3.2 取出全部商品类别
result3_add = bj_data_rdd.map(lambda x:x['category']).distinct()
print("北京售卖的商品类型：", result3_add.collect())

输出结果：
城市销售额排名为： [(‘北京’, 91556), (‘杭州’, 28831), (‘天津’, 12260), (‘上海’, 1513), (‘郑州’, 1120)]

全部城市售卖的商品类型： [‘平板电脑’, ‘家电’, ‘书籍’, ‘手机’, ‘电脑’, ‘家具’, ‘食品’, ‘服饰’]

北京售卖的商品类型： [‘平板电脑’, ‘家电’, ‘书籍’, ‘手机’, ‘电脑’, ‘家具’, ‘食品’, ‘服饰’]