安装Hive3.1.1

安装Hive3.1.1

前提：安装了hadoop，本例是伪分布模式下的hadoop，hadoop伪分布模式安装见Hadoop:单节点设置
1、下载hive，将安装包放置到服务器目录，进入你放置hive安装包的目录，解压。

cd /opt
tar -zxvf apache-hive-3.1.1-bin.tar.gz

2、配置hive环境变量。

echo "export HIVE_HOME=$PWD/apache-hive-3.1.1-bin" > /etc/profile.d/hive.sh
echo "PATH=$PATH:$HIVE_HOME/bin" >> /etc/profile.d/hive.sh
source /etc/profile

3、进入conf目录生成hive-site.xml配置文件

cd conf/
cp hive-default.sh.template hive-site.sh

4、配置hive-site.xml

<!--默认数据库仓库的位置，存储表的元数据-->
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

同时创建该目录

mkdir -p /user/hive/warehouse

5、初始化数据库

schematool -initSchema -dbType derby

6、运行hive

hive

会报错：

Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: **Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]**
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3003)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2931)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2806)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1460)
	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
	at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5098)
	at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
	at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3210,96,"file:/opt/apache-hive-3.1.1-bin/conf/hive-site.xml"]
	at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
	at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
	at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
	at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
	at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
	at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
	at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
	at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
	at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3257)
	at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3063)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2986)
	... 15 more

非法字符，hive-site.xml的3210行有错，

vim +3210 conf/hive-site.xml

3206   <property>
3207     <name>hive.txn.xlock.iow</name>
3208     <value>true</value>
3209     <description>
3210       Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for&#8;transactional tables.  This ensures that inserts (w/o overwrite) running concurrently
3211       are not hidden by the INSERT OVERWRITE.
3212     </description>
3213   </property>

删除
7、运行hive
报错：

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
	at org.apache.hadoop.fs.Path.initialize(Path.java:259)
	at org.apache.hadoop.fs.Path.<init>(Path.java:217)
	at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:707)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:624)
	at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:588)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:747)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
	at java.net.URI.checkPath(URI.java:1823)
	at java.net.URI.<init>(URI.java:745)
	at org.apache.hadoop.fs.Path.initialize(Path.java:256)
	... 12 more

URI路径问题，在配置文件hive-site.xml里面将${system:java.io.tmpdir} 全部替换为/opt/apache-hive-3.1.1-bin/tmp 将{system:user.name} 全部替换为 {user.name}
vim命令行模式输入以下命令全局替换。

:1,$s/${system:java.io.tmpdir}/\/opt\/apache-hive-3.1.1-bin\/tmp/g
:1,$s/{system:user.name}/{user.name}/g

8、运行hive

hive

成功！以上练习hive环境搭配hadoop的伪分布模式，使用的是Derby数据库，只能提供有限的，单进程的存储服务，例如用户不可以执行2个并发的hive命令行界面。个人计算机上或者某些开发任务上使用的话没有问题，对于集群来说，需要使用MySQL或者类似的关系型数据库。
9、其他问题
进入hive后使用hql报错：

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

不能初始化元数据库，需要配置第四步，指定默认数据库仓库的位置，删掉你之前工作目录（即在哪个目录运行了hive这个命令）中的metastore_db，并重新初始化。

rm -rf metastore_db
schematool -initSchema -dbType derby

10、设置mysql为元数据库

• 1.在mysql中创建hive元数据库：

create database hive_db;

• 2.修改hive-site.xml中一下属性：

    <property>
    	<!--用户名-->
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
	   	<!--密码-->
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
    </property>
   <property>
   		<!--mysql数据库连接-->
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.1.68:3306/hive_db?useSSL=TRUE</value>
    </property>
    <property>
    	<!--mysql驱动程序-->
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

• 3.复制mysql的驱动程序到hive/lib下面。
• 4.初始化数据库:

schematool -dbType mysql -initSchema

• 5.运行hive