这次的hive搭建过程是在是累。
版本2.3.1
实际使用2.3.5
两个不同的hive版本虽有一丝丝的差别,但是在配置hive-site.xml的过程却艰难往复。
hive的搭建需要MySQL作为前提。
记录一下这次搭建的过程以及一些坑。
先介绍坑吧。
坑1:版本使用错误,原先是需要2.3.1版本的,但是我看官网上没有,而且提供的资料是2.3.5版本的。所以我就尝试。
在多次失败选择MySQL作为库之后,只能找老师帮助,在老师的帮助下发现是hive版本的问题。
坑2:MySQL用户名和密码正确,也可以连接上navicat但是无法进行初始化。这里可能是source没有生效,但是source ./filepath/也莫名其妙的不能使用。可能是我知识掌握不够牢固,但是计算机编程这东西,总是会有各种各样的奇怪的事情发生。
这冰冷的代码甚是无情!
其他的感觉还好。
介绍一下hive-2.3.1的搭建:
1 安装MySQL
过程在这:https://blog.csdn.net/withanything/article/details/107468521
(MySQL开启远程连接权限)
2启动两项服务:
service mysqld start 启动mysql服务
start-all.sh 启动Hadoop服务
顺便使用jps查看Hadoop的进程是否正确:
namenode
secondrynamenode
resourcemanager
datanode
jps
差不多就这些。
3配置方面
3.1配置
进入到hive的conf文件夹中
[root@ahu01 conf]# pwd
/opt/apps/hive-2.3.1/conf
1) cp hive-env.sh.template hive-env.sh
注意路径
vi hive-env.sh
export HADOOP_HOME=/opt/apps/hadoop-2.8.5/
# Hive Configuration Directory can be controlled by:
export HIVE_CONF_DIR=/opt/apps/hive-2.3.1/conf/
2)配置hive-site.xml
<configuration>
<!-- 记录HIve中的元数据信息 记录在mysql中 -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://ahu01:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!-- mysql的用户名和密码 -->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property><property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property><property>
<name>hive.exec.scratchdir</name>
<value>/user/hive/tmp</value>
</property>
<property>
<name>hive.querylog.location</name>
<value>/user/hive/log</value>
</property>
<!-- 客户端远程连接的端口 -->
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
<property>
<name>hive.server2.thrift.bind.host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hive.server2.webui.host</name>
<value>0.0.0.0</value>
</property>
<!-- hive服务的页面的端口 -->
<property>
<name>hive.server2.webui.port</name>
<value>10002</value>
</property>
<property>
<name>hive.server2.long.polling.timeout</name>
<value>5000</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property><property>
<name>hive.execution.engine</name>
<value>mr</value>
</property>
</configuration>3) vi /opt/apps/hadoop-2.8.5/etc/hadoop/core-site.xml
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property><property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property><property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
3) vi /opt/apps/hadoop-2.8.5/etc/hadoop/core-site.xml
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property><property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property><property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
4 上传mysql驱动包
上传一个mysql的驱动包到hive 的lib包下
mysql-connector-java-5.1.39.jar
初始化
cd /opt/apps/hive-2.3.1/
5 启动
./bin/schematool -initSchema -dbType mysql
[root@linux01 hive-2.3.1]# bin/hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apps/hive-2.3.1/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/apps/hive-2.3.1/lib/hive-common-2.3.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
6 hive连接方式
6.1 本地客户端连接
本地测试默认用于学习和测试使用, 实际生产中一般远程连接方式
[root@linux01 hive-2.3.1]# bin/hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apps/hive-2.3.1/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/apps/hive-2.3.1/lib/hive-common-2.3.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
4.2 远程连接hiveserver2
前台启动 bin/hiveserver2
[root@linux01 hive-2.3.1]# bin/hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apps/hive-2.3.1/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/opt/apps/hive-2.3.1/lib/hive-common-2.3.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
后台启动 bin/hiveserver2 &
查看10000 端口是否使用
[root@linux01 ~]# netstat -nltp | grep 10000
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 55020/java
4.2.1 beeline
[root@linux01 ~]# beeline
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apps/hive-2.3.1/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apps/hadoop-2.8.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 2.3.1 by Apache Hive
beeline> !connect jdbc:hive2://linux01:10000
Connecting to jdbc:hive2://linux01:10000
Enter username for jdbc:hive2://linux01:10000: root
Enter password for jdbc:hive2://linux01:10000: 回车
Connected to: Apache Hive (version 2.3.1)
Driver: Hive JDBC (version 2.3.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://linux01:10000>
结果
0: jdbc:hive2://linux01:10000> show databases ;
+----------------+
| database_name |
+----------------+
| db_doit15 |
| db_doit17 |
| default |
+----------------+