AttributeError: module ‘pyspark.rdd‘ has no attribute ‘V‘

目录

报错描述

原因

解决方案

再次运行

结语

---

报错描述

AttributeError: module 'pyspark.rdd' has no attribute 'T'

AttributeError: module 'pyspark.rdd' has no attribute 'V'

运行下方代码时发生错误：

from pyspark import SparkConf, SparkContext

conf = SparkConf().setAppName("WordCountHelloWorld")
sc = SparkContext(conf=conf)

rdd = sc.parallelize([1,2,3,4,5,6,7,8,9],3)
rdd.glom().collect()

错误提示(这里只展示部分错误提示)：

2022-11-15 10:29:27,271 ERROR executor.Executor: Exception in task 2.0 in stage 0.0 (TID 2)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/opt/module/spark-3.2.0/python/lib/pyspark.zip/pyspark/worker.py", line 601, in main
    func, profiler, deserializer, serializer = read_command(pickleSer, infile)
  File "/opt/module/spark-3.2.0/python/lib/pyspark.zip/pyspark/worker.py", line 71, in read_command
    command = serializer._read_with_length(file)
  File "/opt/module/spark-3.2.0/python/lib/pyspark.zip/pyspark/serializers.py", line 160, in _read_with_length
    return self.loads(obj)
  File "/opt/module/spark-3.2.0/python/lib/pyspark.zip/pyspark/serializers.py", line 430, in loads
    return pickle.loads(obj, encoding=encoding)
AttributeError: module 'pyspark.rdd' has no attribute 'T'

......

原因

spark版本和python库中pyspark版本不兼容

解决方案

查看当前spark版本：

(pyspark) [hadoop@hadoop-001 ~ ]$SPARK_HOME/bin/pyspark

卸载pyspark库

pip uninstall pyspark

安装对应版本的pyspark

pip install pyspark==3.2.0

再次运行

成功！

结语

比起在python中调用pyspark，最好当然还是直接进入spark官方提供的 bin/pyspark 来运行代码

但是笔者因为要在windows下通过ssh配置linux远程解释器，因此要搭建好python环境，因此有了上述问题。