spark on yarn与idea直连操作sql遇到的一些问题
先上代码
SparkConf conf = new SparkConf().setAppName("programdept").setMaster("yarn-client")
.set("yarn.resourcemanager.hostname", "10.100.200.11")
// .set("spark.executor.instances","1")
// .set("spark.executor.memory","3500M")
.setJars(new String[]{"D:\\code\\sparkonyarn\\out\\artifacts\\sparkonyarn.jar"})
.set("spark.yarn.preserve.staging.files","false")
.set("spark.executor.extraClassPath"," /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/hive/lib/*");
JavaSparkContext sparkContext = new JavaSparkContext(conf);
HiveContext hiveContext = new HiveContext(sparkContext.sc());
// hiveContext.clearCache();
hiveContext.sql("use oracle_table");