Centos7使用scala从外部MySQL数据库加载数据集出现ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting..

环境配置:使用的是Centos7.4版本的虚拟机,然后hive配置的是远程模式(master节点当作客户端,slave节点当作hive端),slave2节点安装的是MySQL数据库;spark版本为1.6.3,scala版本为2.10.6
目的:在spark-shell中使用scala加载外部MySQL数据库中的菜品数据,然后做数据分析工作。
出现的问题:在master节点上访问MySQL数据库中的数据库的时候出现问题。

ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most r in stage 0.0 (TID 3, slave): java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
21/12/16 20:52:11 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, slave2): java.lang.ClassNotFoundExcr
	at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:38)
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply
	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.<init>(JDBCRDD.scala:347)
	at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(JDBCRDD.scala:339)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

21/12/16 20:52:12 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most r in stage 0.0 (TID 3, slave): java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
	at org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)

scala代码:

// 从外部数据库(MySQL加载菜品数据)
import org.apache.spark.sql.SQLContext
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val url = "jdbc:mysql://slave2:3306/MealData"
val mealDF=sqlContext.read.format("jdbc").options(Map("url"->url,"user"->"root","password"->"123456","dbtable"->"meal_list")).load()
// 生成菜品数据
val mealsMap=mealDF.select("mealID","meal_name").map(row=>(row.getString(0),row.getString(1))).collect.toMap

在这里插入图片描述
在这里插入图片描述

一直显示没有连接上MySQL服务。
解决办法:将MySQL的链接jar报复制到spark的lib目录下,类似于hive的远程模式中要把MySQL连接jar包复制到hive的lib目录下。
然后再在配置文件中spark-defaults.conf文件中添加如下配置项

spark.executor.extraClassPath   /usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/mysql-connector-java-5.1.27-bin.jar
spark.driver.extraClassPath    /usr/local/src/spark-1.6.3-bin-hadoop2.6/lib/mysql-connector-java-5.1.27-bin.jar

在这里插入图片描述
然后退出 spark-shell 重新进入在执行代码:
在这里插入图片描述

  • 1
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值