配置pyspark环境

环境搭建
jdk 1.8配置 https://www.cnblogs.com/nojacky/p/9497724.html
安装spark以及hadoop https://zhuanlan.zhihu.com/p/103465541
python 环境3.7

报错
pyspark在windows下java.net.SocketException: Connection reset by peer 错误
解决方案:
https://blog.csdn.net/dongtest/article/details/84829602

pyspark在windows加载数据集 训练模型出现 以下错误
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)

在stackoverflow.com上找到的解决办法
修改spark/python/pyspark目录下worker.py文件
修改 def process():为以下内容

    def process():
        iterator = deserializer.load_stream(infile)
        serializer.dump_stream(func(split_index, iterator), outfile)
        for obj in iterator:
            pass

然后将pyspark目录打包为zip 替换掉spark/python/lib/目录下对应的pyspark.zip文件