hive on spark安装

Hive on spark安装

1.      下载apache-hive-2.0.0-bin.tar.gz，安装。（尽量安装和hive相对应的版本spark和hadoop）

2.      编译spark（不带hive的）

./make-distribution.sh--name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"

见：https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

将编译后的spark下的lib下的 spark-assembly jar 拷贝到hive的lib下。

3.      hive-env.sh配置：

exportHIVE_AUX_JARS_PATH=/home/hadoop/app/apache-hive-2.0.0-bin/lib

 

exportHADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.6.0

export HIVE_CONF_DIR=/home/hadoop/app/apache-hive-2.0.0-bin/conf

exportHIVE_HOME=/home/hadoop/app/apache-hive-2.0.0-bin

export JAVA_HOME=/usr/java/jdk1.7.0_79

4.      hive-site配置：

<!--在hdfs上hive数据存放目录，启动hadoop后需要在hdfs上手动创建 -->                   <property>              <name>hive.metastore.schema.verification</name>              <value>false</value>               </property>                   <!--默认 metastore 在本地，添加配置改为非本地          <property>                    <name>hive.metastore.local</name>                    <value>false</value>          </property>-->                   <property>                    <name>hive.metastore.uris</name>                    <value>thrift://bihdp01:9083</value>                    <description>Thrift uri for the remote metastore. Used by metastore client to connect to remote metastore.</description>          </property>                            <property>                    <name>hive.metastore.warehouse.dir</name>                    <value>/hive/warehouse</value>          </property>          <!--通过jdbc协议连接mysql的hive库 -->          <property>                    <name>javax.jdo.option.ConnectionURL</name>                    <value>jdbc:mysql://bihdp01:3306/hiveto?createDatabaseIfNotExist=true</value>                    <description>JDBC connect string for a JDBC metastore</description>          </property>          <!--jdbc的mysql驱动 -->          <property>                    <name>javax.jdo.option.ConnectionDriverName</name>                    <value>com.mysql.jdbc.Driver</value>                    <description>Driver class name for a JDBC metastore</description>          </property>          <!--mysql用户名 -->          <property>                    <name>javax.jdo.option.ConnectionUserName</name>                    <value>root</value>                    <description>username to use against metastore database</description>          </property>          <!--mysql用户密码 -->          <property>                    <name>javax.jdo.option.ConnectionPassword</name>                    <value>*********</value>                    <description>password to use against metastore database</description>          </property>                                              <!-- 设置为false，查询将以运行hiveserver2进程的用户运行-->                   <property>                    <name>hive.server2.enable.doAs</name>                    <value>ture</value>          </property>          <property>                    <name>hive.server2.thrift.bind.host</name>                    <value>bihdp01</value>          </property>          <property>                    <name>hive.server2.thrift.port</name>                    <value>10000</value>          </property>                            <property>                    <name>hive.exec.parallel</name>                    <value>true</value>          </property>          <property>                    <name>hive.exec.dynamic.partition.mode</name>                    <value>strict</value>          </property>          <property>                    <name>hive.exec.compress.intermediate</name>                    <value>true</value>          </property>          <!-- 配置hive 的web 页面访问的接口hwi , 主机 端口  war包的路径-->          <property>     <name>hive.hwi.listen.host</name>     <value>bihdp01</value>   </property>     <property>     <name>hive.hwi.listen.port</name>     <value>9999</value>   </property>   <property>     <name>hive.hwi.war.file</name>     <value>lib/hive-hwi-1.2.1.war</value>   </property>         <property> <name>spark.eventLog.enabled</name> <value>true</value> </property> <!—hdfs目录存在--> <property> <name>spark.eventLog.dir</name> <value>hdfs:///hive_on_sparklogs</value> </property>   <property> <name>spark.executor.memory</name> <value>512m</value> </property>   <property> <name>spark.serializer</name> <value>org.apache.spark.serializer.KryoSerializer</value> </property> </configuration>

5.      启动hive

sethive.execution.engine=spark;

set spark.master=yarn-cluster;(目前只有这个测试有效)

（也可以配置在hive-site.xml中）

 

 

遇到问题：http://91r.net/ask/31228420.html