qq_51697761的博客

进行数据清洗：transformer.pyfrom pyspark import SparkContextdef main(): sc=SparkContext(appName="tranformer_counter") raw=sc.textFile("file:///root/spark/transformer.csv") rdd=raw.map(lambda x:x.split(",")[1:3])\ .map(lamb...

structured_streaming.py代码from pyspark.sql import SparkSessionfrom pyspark.sql.functions import splitfrom pyspark.sql.functions import explodeimport sysif __name__=="__main__": spark=SparkSession\ .builder\ .appNam...

搭建集群重新开一个端口还是在spark-master里面运行一直在不停的统计数据job.py代码：# import findspark# findspark.init()from pyspark import SparkContextfrom pyspark.streaming import StreamingContextdef updateFunc(values,state): cnt = 0 for v in values: ...

题目：代码：fault_detect.pyimport sysfrom pyspark import SparkContextfrom pyspark.streaming import StreamingContextdef detect(new_values,fault_state): for value in new_values: if value<1.0: fault_state=1.0 elif val

socket_wordcpunt.py代码：import sysfrom pyspark import SparkContextfrom pyspark.streaming import StreamingContext if __name__=="__main__": sc=SparkContext(appName="PythonStreamingNetworkWordCount") sc.setLogLevel("ERROR") ssc=StreamingConte

spark集群搭建

fileStream.py代码from pyspark.streaming import StreamingContextfrom pyspark import SparkContextsc=SparkContext(appName='test straming')sc.setLogLevel("ERROR")ssc=StreamingContext(sc,2)line=ssc.textFileStream("file:///root/recruit/data")rdd=line.m

socket流词频统计2、新开一个worker1窗口3、在maser窗口输入词频，以“ ”空格分开

搭建集群：docker load --input spark.tardocker-compose up -ddocker psdocker exec -it spark-master bash #在主机端口jps#查看在主机窗口：重新开一个窗口worker1：结果：socker-server.py代码# from concurrent.futures import thread# from distutils.log imp...