记录Pyspark使用中遇到一次填坑过程,得到个血泪的教训啊
环境
Spark 2.3.0
集群模式:yarn
Python 3.x
CentOS 7
问题1
在通过spark-submit提交任务时,出现首次报错
task 0.3 in stage 73.0 (TID 3953, h2, executor 1): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data/yarn/usercache/root/appcache/application_1596705985506_0151/container_1596705985506_0151_01_000002/pyspark.zip/pyspark/worker.py", line 175, in main
("%d.%d" % sys