Spark2.4.2报错总结

最新推荐文章于 2023-08-27 11:59:38 发布

浅汐王

最新推荐文章于 2023-08-27 11:59:38 发布

阅读量931

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/qq_32252917/article/details/103691823

版权

这篇博客总结了在Spark 2.4.2中遇到的三个常见错误：1) Spark Shell无法连接到Hive metastore，原因是缺少mysql-connector jar包。解决方法是在提交作业时指定--jars参数；2) Spark SQL启动报错，driver端缺乏mysql驱动。通过添加driver的classpath路径成功解决；3) IDEA中RDD转DataFrame出现jar包冲突，解决办法是排除冲突的janino和compiler低版本jar包。

摘要由CSDN通过智能技术生成

报错一:

背景

启动spark-shell后查询hive中的表信息，报错

$SPARK_HOME/bin/spark-shell
spark.sql("select * from student.student ").show()

日志

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
    at org.apache.hadoop.hive.metastore.RetryingMetaSto

    Caused by: org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException:
    The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the 
    CLASSPATH. Please check your CLASSPATH specification, 
    and the name of the driver.

原因:

spark访问存放hive的Metastore的mysql数据库时，没有连接成功，因为spark没有没有mysql-connecter的jar包，即mysql驱动

解决:

有人说，那就直接把jar包cp到$SPARK_HOME/jars下呗，不好意思，这样再生产中是绝对不行的，并不是所有spark程序都会用到mysql驱动，所以我们要在提交作业时指定--jars，多个jar包用逗号分隔注(我的mysql版本是5.1.73)

[hadoop@hadoop003 spark]$ spark-shell --jars ~/softwares/mysql-connector-java-5.1.47.jar 
19/05/21 08:02:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://hadoop003:4040
Spark context available as 'sc' (master = local[*], app id = local-1558440185051).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.2
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.

scala>  spark.sql("select * from student.student ").show()
19/05/21 08:04:42 WARN DataNucleus.General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/home/hadoop/app/spark-2.4.2-bin-hadoop-2.6.0-cdh5.7.0/jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/home/hadoop/app/spark/jars/datanucleus-core-3.2.10.jar."
19/05/21 08:04:42 WARN DataNucleus.General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/home/hadoop/app/spark/jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/home/hadoop/app/spark-2.4.2-bin-hadoop-2.6.0-cdh5.7.0/jars/datanucleus-api-jdo-3.2.6.jar."
19