PySpark :EOFException occurred while reading the port number from pyspark.daemon‘s stdout

1.pychrm远程连接服务器

1.点击pycharm 中file 新建项目

2.点击增加解释器Add Interpreter

3.接下来就是登录自己的服务器外网账号一路next

注意：我的prcharm是2022版的，所以直接在Add Interpreter这里可以直接选择，不同的版本不同位置，注意区分

点击右下角红框位置

可以设置解释器 

这里可以看到所有的版本号

环境配好运行如下代码：

# coding:utf8
from pyspark import SparkConf, SparkContext

if __name__ == '__main__':
    print(12121)
    conf = SparkConf().setAppName("WordCountHelloWorld").setMaster("local[*]")
    # 通过SparkConf对象构建SparkContext对象
    sc = SparkContext(conf=conf)

    # 需求 : wordcount单词计数, 读取HDFS上的words.txt文件, 对其内部的单词统计出现 的数量
    # 读取文件

    file_rdd = sc.textFile("hdfs://leida102:8020/input/words.txt")
    #file_rdd = sc.parallelize(['hadoop hadoop', 'hadoop spark', 'spark spark', 'spark', 'flink', 'flink', 'flink', 'flink', 'flink'])
    #file_rdd = sc.textFile("../data/input/words.txt")


    # 将单词进行切割, 得到一个存储全部单词的集合对象
    words_rdd = file_rdd.flatMap(lambda line: line.split(" "))

    # 将单词转换为元组对象, key是单词, value是数字1
    words_with_one_rdd = words_rdd.map(lambda x: (x, 1))

    # 将元组的value 按照key来分组, 对所有的value执行聚合操作(相加)
    result_rdd = words_with_one_rdd.reduceByKey(lambda a, b: a + b)

    # 通过collect方法收集RDD的数据打印输出结果
    print(result_rdd.collect())

发现报错：

Caused by: org.apache.spark.SparkException: 
Error from python worker:
  Traceback (most recent call last):
    File "<frozen runpy>", line 189, in _run_module_as_main
    File "<frozen runpy>", line 112, in _get_module_details
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 51, in <module>
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/context.py", line 30, in <module>
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/accumulators.py", line 97, in <module>
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 71, in <module>
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 209, in <module>
    File "/opt/package/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 172, in _make_cell_set_template_code
  TypeError: code() argument 13 must be str, not int
PYTHONPATH was:
  /opt/package/spark/python/lib/pyspark.zip:/opt/package/spark/python/lib/py4j-0.10.9-src.zip:/opt/package/spark/jars/spark-core_2.12-3.0.0.jar:/./tmp/pycharm_project_688/:/home/xinjie/.pycharm_helpers/pycharm_matplotlib_backend:/home/xinjie/.pycharm_helpers/pycharm_display
org.apache.spark.SparkException: EOFException occurred while reading the port number from pyspark.daemon's stdout
	at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:217)
	at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:132)
	at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:105)
	at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:119)
	at org.apache.spark.api.python.BasePythonRunner.compute(PythonRunner.scala:131)
	at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:103)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

网上找好多资料，有说是版本不兼容

但是我直接在服务器上的anaconda 创建的pysaprk虚拟环境下直接运行我提交的代码时

有意思的发现可以成功执行

后面在代码中直接指定环境变量解决了

代码如下：

# coding:utf8
from pyspark import SparkConf, SparkContext
import os
PYSPARK_PYTHON="/home/xinjie/anaconda3/envs/pyspark/bin/python"
# 设置环境变量
os.environ['PYSPARK_PYTHON']= PYSPARK_PYTHON
SPARK_HOME="/opt/package/spark/"
os.environ['SPARK_HOME']= SPARK_HOME

if __name__ == '__main__':
    print(12121)
    conf = SparkConf().setAppName("WordCountHelloWorld").setMaster("local[*]")
    # 通过SparkConf对象构建SparkContext对象
    sc = SparkContext(conf=conf)

    # 需求 : wordcount单词计数, 读取HDFS上的words.txt文件, 对其内部的单词统计出现 的数量
    # 读取文件

    file_rdd = sc.textFile("hdfs://leida102:8020/input/words.txt")
    #file_rdd = sc.parallelize(['hadoop hadoop', 'hadoop spark', 'spark spark', 'spark', 'flink', 'flink', 'flink', 'flink', 'flink'])
    #file_rdd = sc.textFile("../data/input/words.txt")


    # 将单词进行切割, 得到一个存储全部单词的集合对象
    words_rdd = file_rdd.flatMap(lambda line: line.split(" "))

    # 将单词转换为元组对象, key是单词, value是数字1
    words_with_one_rdd = words_rdd.map(lambda x: (x, 1))

    # 将元组的value 按照key来分组, 对所有的value执行聚合操作(相加)
    result_rdd = words_with_one_rdd.reduceByKey(lambda a, b: a + b)

    # 通过collect方法收集RDD的数据打印输出结果
    print(result_rdd.collect())

执行结果如下

/home/xinjie/anaconda3/envs/pyspark/bin/python /./tmp/pycharm_project_688/rdd/HelloWord.py 
12121
2023-08-07 15:05:32,237 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2023-08-07 15:05:33,121 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
2023-08-07 15:05:33,121 WARN util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
2023-08-07 15:05:33,123 WARN util.Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
[('hadoop', 1), ('hello', 3), ('spark', 1), ('flink', 1)]

Process finished with exit code 0

原因分析：安装anaconda的时候，我并没有配置anaconda的环境变量，进入服务器自己创建的pyspark虚拟环境下可以正常执行代码，我估计pycharm并没有识别到pyspark的环境变量，所以直接在代码中把环境变量直接写死。