编写Structure streaming程序

步骤：
1.导入pyspark模块
2.创建SparkSession对象
3.创建输入数据源
4.定义流计算过程
5.启动流计算并输出结果

from pyspark.sql import SparkSession
from pyspark.sql.functions import split
from pyspark.sql.functions import explode#炸成一行一个单词

if__name__==“main”:
spark = SparkSession.builder.appName(“StructuredNetworkWordCount”).getOrCreate()
spark.sparkContext.setLogLevel(‘WARN’)#设置显示级别，只有WARN的过程才被显示

lines = spark.readStream.format(“socket”).option(“host”,“localhost”).option(“port”,9999).load()

words = lines.select(explode(split(lines.value," "))).alias(“word”)#过来的是DataFrame，一行单词过来了，通过空格拆分成数组或是列表，然后炸成一个一个的单词，得到一个新的DataFrame，每一行就是一个单词，alias是重新命名一列为word
wordCounts = words.groupBy(“word”).count()

query =wordCounts.writeStream.outputMode(“complete”).format(“console”).trigger(processingTime = “8 seconds”).start()
query.awaitTermination()

先启动HDFS
cd /usr/local/hadoop
sbin/start-dfs.sh
新建终端，在这输入数据就行
nv -lk 9999
新建终端客户端
cd /usr/local/spark/mycode/structuredstreaming/
/usr/local/spark/bin/spark-submit
StruturedNetworkWordCount.py