spark jar包方式运行hive步骤

13 篇文章 0 订阅
1,用maven install 打包
2,配置好下面的脚本,其中--driver-class-path要选择mysql的jar包所在的目录,另外有博客说可以在spark-env指定
export SPARK_CLASSPATH=$SPARK_HOME/lib/mysql-connector-java-5.0.8-bin.jar
-----注意配置了--driver-class-path 就不能再配spark-env不然,启动会报错:
Exception in thread "main" org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.

----但是用spark-env方式配置,运行还是报错找不断mysql驱动:

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

run-sql.sh内容:
#!/bin/sh
FWDIR="$SPARK_HOME"
$FWDIR/bin/spark-submit \
--class zfh.spark.study.hive.HiveDataSource \
--num-executors 3 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 3 \
--files /home/zfh/apache/hive-2.0.0/conf/hive-site.xml \
--driver-class-path /home/zfh/apache/hive-2.0.0/lib/mysql-connector-java-5.0.8-bin.jar \
/home/zfh/runjar/spark-study-java-0.0.1.jar \
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值