Centos7 Hive 部署

Centos7 Hive 部署

1、准备部署环境

1、准备hadoop 环境
部署步骤参照 hadoop安装步骤
2、安装mysql 数据库
安装步骤参照 mysql安装步骤
3、准备mysql jdbc 驱动包
4、准备与hadoop 版本相互兼容的 安装包 https://hive.apache.org/downloads.html
下载 hive 安装包

2、部署

1、上传安装包至 服务器上

2、解压 安装包 并移动至 opt 目录下

# 解压至  /opt 目录下
[root@syq-jtj-jzjxyth-yycx3 software]# tar -zxvf apache-hive-2.1.1-bin.tar.gz -C /opt/
apache-hive-2.1.1-bin/LICENSE
apache-hive-2.1.1-bin/NOTICE
apache-hive-2.1.1-bin/README.txt
apache-hive-2.1.1-bin/RELEASE_NOTES.txt
..................................
将文件夹重命名 为 hive 
[root@syq-jtj-jzjxyth-yycx3 opt]# mv apache-hive-2.1.1-bin hive
[root@syq-jtj-jzjxyth-yycx3 opt]# ls
hadoop  hive

3、将mysql jar包驱动 copy 到 /opt/hive/lib 目录下

[root@syq-jtj-jzjxyth-yycx3 bin]# cp /software/mysql-connector-java-5.1.34.jar /opt/hive/lib/
[root@syq-jtj-jzjxyth-yycx3 bin]# 

4、配置环境变量

[root@syq-jtj-jzjxyth-yycx3 hive]# vim /etc/profile

#Hive 
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin

source 环境变量 
[root@syq-jtj-jzjxyth-yycx3 hive]# source /etc/profile

3、在HDFS 上创建 hive 存储目录

创建  hive 存储目录
[root@syq-jtj-jzjxyth-yycx3 hadoop]# cd /opt/hadoop/bin/
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -mkdir -p /user/hive/warehouse
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -mkdir -p /user/hive/tmp
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -mkdir -p /user/hive/log
给创建的目录授权 
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -chmod -R 777 /user/hive/warehouse
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -chmod -R 777 /user/hive/tmp
[root@syq-jtj-jzjxyth-yycx3 bin]# ./hdfs dfs -chmod -R 777 /user/hive/log

4、修改hive 配置文件

①、修改hive-env.sh

将 config 下 hive-env.sh.template 复制一份  hive-env.sh
[root@syq-jtj-jzjxyth-yycx3 conf]# cp hive-env.sh.template hive-env.sh
编辑 hive-env.sh   下拉到最后加上 以下配置
##hadoop的home
HADOOP_HOME=/opt/hadoop     
#hive的home    
export HIVE_CONF_DIR=/opt/hive/conf
export HIVE_AUX_JARS_PATH=/opt/hive/lib

②、修改hive-size.xml

将 config 下 hive-default.xml.template 复制一份  hive-site.xml
[root@syq-jtj-jzjxyth-yycx3 conf]# cp hive-default.xml.template hive-site.xml
打开  hive-site.xml 文件 

1、搜索 hive.metastore.warehouse.dir    修改成hdfs 上的创建的路径
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>location of default database for the warehouse</description>
  </property>

2、搜索hive.exec.scratchdir,将该name对应的value修改为/user/hive/tmp
<property>  
    <name>hive.exec.scratchdir</name>  
    <value>/user/hive/tmp</value>  
</property>

3、搜索hive.querylog.location,将该name对应的value修改为/user/hive/log/hadoop
<property>
    <name>hive.querylog.location</name>
    <value>/user/hive/log/hadoop</value>
    <description>Location of Hive run time structured log file</description>
</property>


4、搜索javax.jdo.option.connectionURL,将该name对应的value修改为MySQL的地址
必须先创建 mysql 数据库 然后配置自己的数据库连接地址
jdbc:mysql://syq-jtj-jzjxyth-yycx3:3306/hive?createDatabaseIfNotExist=true
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://syq-jtj-jzjxyth-yycx3:3306/hive?createDatabaseIfNotExist=true</value>
    <description>
      JDBC connect string for a JDBC metastore.
      To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
      For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
    </description>
</property>

5、搜索javax.jdo.option.ConnectionDriverName，将该name对应的value修改为MySQL驱动类路径
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
</property>

6、搜索javax.jdo.option.ConnectionUserName，将对应的value修改为MySQL数据库登录名
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>

7、搜索javax.jdo.option.ConnectionPassword，将对应的value修改为MySQL数据库的登录密码
 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
    <description>password to use against metastore database</description>
  </property>

8、指定hive 运行的tmp目录(一定要改否则会报错)
创建 tmp 目录
[root@syq-jtj-jzjxyth-yycx3 /]# mkdir /usr/hive
[root@syq-jtj-jzjxyth-yycx3 /]# mkdir /usr/hive/tmp
 
并在 hive-site.xml 中修改
把 ${system:java.io.tmpdir} 改成 /usr/hive/tmp
把 {system:user.name} 改成 {user.name}

5、hive 数据初始化

[root@syq-jtj-jzjxyth-yycx3 hive]# pwd
/opt/hive
[root@syq-jtj-jzjxyth-yycx3 hive]# bin/schematool -initSchema -dbType mysql
which: no hbase in (/usr/jdk1.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/hadoop/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://syq-jtj-jzjxyth-yycx3:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       hive
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException : Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
SQL Error code: 0
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
[root@syq-jtj-jzjxyth-yycx3 hive]# bin/schematool -initSchema -dbType mysql
which: no hbase in (/usr/jdk1.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/hadoop/bin:/root/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://10.75.8.36:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       hive
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.mysql.sql
Initialization script completed
schemaTool completed
[root@syq-jtj-jzjxyth-yycx3 hive]# 

6、测试 hive

[root@syq-jtj-jzjxyth-yycx3 hive]# hive
which: no hbase in (/usr/jdk1.8/bin:/usr/jdk1.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/hadoop/bin:/root/bin:/opt/hadoop/bin:/opt/hive/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/hive/lib/hive-common-2.1.1.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive> show databases;
OK
default
Time taken: 1.019 seconds, Fetched: 1 row(s)
hive> create database ods;
OK
Time taken: 0.225 seconds
hive> use ods;
OK
Time taken: 0.027 seconds
hive> create table dim_user;
FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified
hive> create table dim_user(id string,name string);
OK
Time taken: 0.247 seconds
hive> insert into dim_user select '1','zhangsan';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20220405233202_654fb824-a2d1-4832-b392-84dd95be5642
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1648891685744_0001, Tracking URL = http://syq-jtj-jzjxyth-yycx3:8088/proxy/application_1648891685744_0001/
Kill Command = /opt/hadoop/bin/hadoop job  -kill job_1648891685744_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2022-04-05 23:32:27,308 Stage-1 map = 0%,  reduce = 0%
2022-04-05 23:32:32,569 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.99 sec
MapReduce Total cumulative CPU time: 1 seconds 990 msec
Ended Job = job_1648891685744_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to directory hdfs://syq-jtj-jzjxyth-yycx3:9000/user/hive/warehouse/ods.db/dim_user/.hive-staging_hive_2022-04-05_23-32-02_668_4247705716408437288-1/-ext-10000
Loading data to table ods.dim_user
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   Cumulative CPU: 1.99 sec   HDFS Read: 4231 HDFS Write: 79 SUCCESS
Total MapReduce CPU Time Spent: 1 seconds 990 msec
OK
Time taken: 31.408 seconds
hive>

测试完毕，查看下 hdfs 上对应的库表数据文件。