spark streaming套接字流流处理-用nc程序创建数据输入源

客户端代码，向服务端发起请求建立连接vim NetworkWordCount.py
#!/usr/bin/env python3
from__future__import print_function
import sys
from pyspark import SparkContext
from pyspark.streaming import StreamingContext

if__name__==“main”:
if len(sys.argv)!=3:
print(“Usage:NetworkWordCount.py<hostname.,port>”,file=sys.stderror)
exit(-1)
sc = SparkContext(appName = “PSWC”)
ssc = StreamingContext(sc,1)
lines = ssc.socketTextStream(sys.arg[1],int(sys.arg[2]))
counts = lines.flatMap(lambda line:line.split(" ")).map(lambda x:(x,1)).reduceByKey(lambda a,b:a+b)
counts.pprint()
ssc.start()
ssc.awaitTermination()
服务端代码，这里用自带的nc程序
$nc -lk 9999#打开一个新的nc窗口,数据输入源，在这个页面里输入，监听窗口自动获取词频统计结果，l是监听listen k是连很多次，断了重连，随便用个空端口就行这里用9999
cd /usr/local/spark/mycode/streaming/socket
/usr/local/spark/bin/spark-submit NetWorkWordCount.py localhost 9999