利用Linux系统中安装的spark来统计:
1.选择目录,并创建一个存放文本的目录,将要处理的文本保存在该目录下以供查找操作:
① cd /usr/local ②mkdir mycode ③ cd mycode ④查看当前新目录: ll
⑤新建文本: vim wordcount.txt (文本内容随机copy一段英文)
[root@node01 mycode]# vim wordcount.txt uded among the most successful influencers in Open Source, The Apache Software Foundation's commitment to collaborative development has long served as a model for producing consistently high quality software that advances the future of open development. https://s.apache.org/PIRA
2.为方便查询文本和其它操作,可以在当前操作节点上复制另一个操作节点,作为它的第二个终端操作窗口:
如:打开:node01------>复制node01 ,然后在复制的节点上去查询之前所创建的目录及文本.
>>>cd /usr/local/mycode/ >>>ll
3.启动spark: 本机spark安装在(/home/mysoft/spark-1.6),以具体路径为准!
① 跳转路径
cd /home/mysoft/spark-1.6
②启动命令: (或者 cd bin ----->pyspark (enter) 亦可)
./bin/pyspark
-------出现spark的正常启动信息即启动成功!
Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ Using Python version 3.5.0 (default, Jul 12 2018 03:34:21) SparkContext available as sc, HiveContext available as sqlContext. >>>
4.加载文件:
>>>textFile = sc.textFile("file:///usr/local/mycode/wordcount.txt") >>> textFile.first()
之后会在屏幕显示之前创建的文本!
注:first()是一个“行动”(Action)类型的操作,会启动真正的计算过程,从文件中加载数据到变量textFile中,并取
出第一行文本,另因为Spark采用了惰性机制,在执行转换操作的时候,即使我们输入了错误的语句,pyspark也不
会马上报错,而是等到执行“行动”类型的语句时启动真正的计算,那个时候“转换”操作语句中的错误就会显示出来:
拒绝连接!
5.统计词频:(继续上述代码输入)
<<<Count = textFile.flatMap(lambda line: line.split(" ")).map(lambda word: (word,1)).reduceByKey(lambda a, b : a + b)
<<<Count.collect()
6.打印结果:
[('development', 1), ('producing', 1), ('among', 1), ('Source,', 1), ('for', 1), ('quality', 1), ('to', 1), ('influencers', 1), ('advances', 1), ('collaborative', 1), ('model', 1), ('in', 1), ('the', 2), ('of', 1), ('has', 1), ('successful', 1), ('Software', 1), ("Foundation's", 1), ('most', 1), ('long', 1), ('that', 1), ('uded', 1), ('as', 1), ('Open', 1), ('The', 1), ('commitment', 1), ('software', 1), ('consistently', 1), ('a', 1), ('development.', 1), ('high', 1), ('future', 1), ('Apache', 1), ('served', 1), ('open', 1), ('https://s.apache.org/PIRA', 1)]