spark中的rdd算子应用

实现词频统计
[‘hadoop,flink,hive’,‘spark,hive,hadoop’,‘spark,hive,python’] 或文件
目标->[(hadoop,2),(flink,1),(spark,2),…]

from pyspark import SparkContext
sc=SparkContext(master='yarn')
#1、将数据转换成rdd对象
rdd1=sc.parallelize(['hadoop,flink,hive','spark,hive,hadoop','spark,hive,python'])
#2、将rdd中每个字符串元素以逗号进行拆分-> ['hadoop', 'flink', 'hive', 'spark', 'hive', 'hadoop', 'spark', 'hive', 'python']
rdd2=rdd1.flatMap(lambda x:x.split(','))
#3、拆分后的rdd构造kv格式的rdd-> [('hadoop', 1), ('flink', 1), ('hive', 1), ('spark', 1), ('hive', 1), ('hadoop', 1), ('spark', 1), ('hive', 1), ('python', 1)]
rdd3=rdd2.map(lambda x:(x,1))
#4、对kv格式的rdd根据key分组，对value值进行累加操作-> [('hive', 3), ('hadoop', 2), ('python', 1), ('flink', 1), ('spark', 2)]
rdd4=rdd3.reduceByKey(lambda x,y:x+y)
print("rdd4->", rdd4.collect())