Spark 1.6 (Java) 问题汇总

最新推荐文章于 2021-04-14 02:40:20 发布

置顶闸北米

最新推荐文章于 2021-04-14 02:40:20 发布

阅读量1.2k

点赞数 2

分类专栏：大数据文章标签：解决方案大数据 Spark

本文链接：https://blog.csdn.net/wc79866/article/details/51564377

版权

大数据专栏收录该内容

2 篇文章 0 订阅

订阅专栏

一、通过SparkSQL读取Oracle时报找不到Oracle JDBC包（java.lang.ClassNotFoundException:oracle.jdbc.driver.OracleDriver）

说明：

程序是本地通过Java将Spark提交给集群运行，但是集群上面没有Oracle JDBC包

解决方法：

官网对于集群运行JDBC的说明：

Troubleshooting

The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java’s DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs.

但是在/etc/profile文件中添加jar路径并没有效果

最终找到方法需要在代码中指定Worker的jar依赖，当然jar包已经上传到每台Worker上,代码如下：

SparkConf conf = new SparkConf().setAppName("Test").setMaster("spark://master2:7077");
		<span style="font-size:14px;color:#ff0000;">conf.set("spark.driver.extraClassPath", "/data/ojdbc14-10.2.0.3.0.jar");
	       conf.set("spark.executor.extraClassPath", "/data/ojdbc14-10.2.0.3.0.jar");</span>

		JavaSparkContext sparkContext = new JavaSparkContext(conf);
		sparkContext.hadoopConfiguration().set("fs.defaultFS", "hdfs://mycluster");
		sparkContext.hadoopConfiguration().set("dfs.nameservices", "mycluster");
		sparkContext.hadoopConfiguration().set("dfs.ha.namenodes.mycluster", "nn1,nn2");
		sparkContext.hadoopConfiguration().set("dfs.namenode.rpc-address.mycluster.nn1", "xx.xx.xx.xxx:8020");
		sparkContext.hadoopConfiguration().set("dfs.namenode.rpc-address.mycluster.nn2", "xx.xx.xx.xxx:8020");
		sparkContext.hadoopConfiguration().set("dfs.client.failover.proxy.provider.mycluster","org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

		
		SQLContext sqlContext = new SQLContext(sparkContext);
		Map<String, String> options = new HashMap<String, String>();
		options.put("url", "jdbc:oracle:thin:mioclub/mioclub3728@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=myrac-scan.mic.com.tw)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=myrac.mic.com.tw)))");
		options.put("dbtable", "mio_login_transfer");
		options.put("driver", "oracle.jdbc.driver.OracleDriver");
		DataFrame jdbcDF = sqlContext.read().format("jdbc"). options(options).load();
		Row a  = jdbcDF.first();
		
		DataFrame d2 = jdbcDF.limit(10);
		d2.rdd().saveAsTextFile("/Test/sql.txt");