spark整合hive配置

spark整合hive配置

1.hive环境配置完成，略~~

2.JAR包

cp {HIVE_HOME}/lib/mysql-connector-java-5.1.44-bin.jar {SPARK_HOME}/jars/
cp {HIVE_HOME}/conf/hive-site.xml {SPARK_HOME}/conf

3.hive-site.xml

编辑{SPARK_HOME}/conf/hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
	<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
	<configuration>
	  <property>
		<name>javax.jdo.option.ConnectionURL</name>
		<value>jdbc:mysql://172.17.XXX.XXX:3306/hive?createDatabaseIfNotExist=true</value>
	　</property>
	　<property>
		<name>javax.jdo.option.ConnectionDriverName</name>
	　　<value>com.mysql.jdbc.Driver</value>
	  </property>
	  <property>
		<name>javax.jdo.option.ConnectionUserName</name>
		<value>root</value>
	  </property>
	  <property>
		<name>javax.jdo.option.ConnectionPassword</name>
		<value>Admin123!</value>
	  </property>
	  <property>
		<name>hive.cli.print.header</name>
		<value>true</value>
	  </property>
	  <property>
		<name>hive.cli.print.current.db</name>
		<value>true</value>
	  </property>

	  <property>
			<name>hive.metastore.warehouse.dir</name>
			<value>/user/hive/warehouse</value>
	  </property>

	</configuration>

4.测试

启动spark-shell：spark-shell

import java.io.File
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
case class Record(key: Int, value: String)
val warehouseLocation = new File("/user/hive/warehouse").getAbsolutePath
val spark = SparkSession.builder().appName("Spark Hive Example").config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()
spark.sql("select * from douyonghou.structure")