Hive远程模式安装报错

先安装hadoop(单机伪分布式模式)：
http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-common/SingleCluster.html

1.下载并解压：
wget mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz

2.java安装并在/etc/profile中设置java环境变量
3.ssh免密登录设置

4.配置hadoop配置文件：

<HADOOP_HOME>/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk.x86_64

<HADOOP_HOME>/etc/hadoop/core-site.xml 
----------------------------------------------------
<property>
  <name>hadoop.tmp.dir</name>
  <value>/hadoop/tmp</value>  ----这里的tmp目录不需要手动建立
</property>
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://127.0.0.1:9000</value>
</property>
------------------------------------------------------

<HADOOP_HOME>/etc/hadoop/hdfs-site.xml
 <property>
  <name>dfs.replication</name>   
  <value>1</value>
 </property>
------------------------------------------------------

<HADOOP_HOME>/etc/hadoop/mapred-site.xml:

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
-------------------------------------------------------
<HADOOP_HOME>/etc/hadoop/yarn-site.xml:

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

5.bin/hdfs namenode -format   -----格式化namenode
   sbin/start-dfs.sh    -----启动dfs

[root@rhel64-64bit sbin]# start-dfs.sh
18/07/05 08:03:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: namenode running as process 19045. Stop it first.
localhost: datanode running as process 19142. Stop it first.
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is 65:7f:dc:67:e8:f2:6e:c7:71:a8:c0:35:21:57:04:3c.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /hadoop/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-rhel64-64bit.out
18/07/05 08:05:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform..
. using builtin-java classes where applicable   -----这里有警告信息

这里的警告原因是：
http://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-common/NativeLibraries.html
The pre-built 32-bit i386-Linux native hadoop library is available as part of the hadoop distribution and is located in
 the lib/native directory. You can download the hadoop distribution from Hadoop Common Releases.
解决办法是下载64bit的library(也可以自己从package里编译)：
http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.6.0.tar
备份原来native目录：mv native native-bak
解压到hadoop的native目录下：
tar -x hadoop-native-64-2.6.0.tar -C  hadoop/hadoop-2.6.5/lib/native/

再次检查native library是否被正确加载：bin/hadoop checknative -a

6.正常启动sbin/start-all.sh结果：
[root@rhel64-64bit sbin]# jps
21252 SecondaryNameNode
21013 NameNode
21526 NodeManager
21559 Jps
21114 DataNode
21391 ResourceManager

 

安装配置hive(远程模式)：
1.下载并解压：
http://hive.apache.org/
https://mirror.bit.edu.cn/apache/hive/hive-2.3.3/
tar -zxvf apache-hive-2.3.3-bin.tar.gz

2.设置hive的环境变量：
export HIVE_HOME=/hive/apache-hive-2.3.3-bin
export PATH=$PATH:$HIVE_HOME/bin

3.HADOOP_HOME/conf/hive-env.sh配置文件：
HADOOP_HOME=/hadoop/hadoop-2.6.5
export HIVE_CONF_DIR=/hive/apache-hive-2.3.3-bin/conf

4.远程模式需要把mysql-connector-java-5.1.46-bin.jar 拷贝到HADOOP_HOME/lib下

5.远程模式的HADOOP_HOME/conf/hive-site.xml文件的配置：

<configuration>

    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root2</value>  ---- mysql的root2用户需要允许远程访问
    </property>

    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>password2</value>   
    </property>

</configuration>

上面的password明文显示不安全，可以参考：Removing Hive Metastore Password from Hive Configuration
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-RemovingHiveMetastorePasswordfromHiveConfiguration

6.打开hive shell：

[root@rhel64-64bit apache-hive-2.3.3-bin]# hive
which: no hbase in (/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin:/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/jre/bin:
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/root/bin:/hadoop/hadoop
-2.6.5/bin:/hadoop/hadoop-2.6.5/sbin:/hive/apache-hive-2.3.3-bin/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hive/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/hive/apache-hive-2.3.3-bin/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> exit
    > ;
Fri Jul 06 07:02:06 GMT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. 
According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option
 isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need
 either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Fri Jul 06 07:02:08 GMT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended.
 According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option
 isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need
 either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

1>SLF4J: Found binding in [jar:file:/hive/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
这个报错原因是jar包重复，删除log4j-slf4j-impl-2.6.2.jar或slf4j-log4j12-1.7.5.jar其中一个即可

2>Fri Jul 06 07:02:06 GMT 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. 
According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option
 isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need
 either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

这个报错是要开启允许non-ssl访问mysql，修改hive-site.xml即可：
 <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>

3>解决上面后，成功进入hive，创建表的时候还是有错误：
hive   > create table t706(tid int);
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
 Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

原因见官网 https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool
Note that the Hive properties to implicitly create or alter the existing schema are disabled by default，
所以手动执行下面初始化命令后，就可以看到在mysql的hive库下创建很多存储metadata的表了.

schematool --help
schematool -dbType mysql -initSchema

[root@rhel64-64bit bin]# schematool -dbType mysql -initSchema
Metastore connection URL:	 jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :	 com.mysql.jdbc.Driver
Metastore connection User:	 root2
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

此时再创建就没问题了：
hive> create table t706(tid int);
OK