1. 前言
本文的安装参照了官方的文档:GettingStarted,将Hive 0.12.0安装在Hadoop 2.4.0上。本文将Hive配置成Server模式,并且使用MySQL作为元数据数据库,远程连接MySQL。
关于Hadoop 2.4.0的安装,请参见《Hadoop-2.4.0分布式安装手册》一文。
2. 约定
本文约定Hadoop被安装在/data/hadoop/current,将Hive 0.12.0的安装到目录/data/hadoop/hive(实际是指向/data/hadoop/hive-0.12.0-bin的软链接)。在实际安装部署时,可以指定为其它目录。
3. 服务端口
10000 | hive.server2.thrift.port,执行hiveserver2时会启动它 |
9083 | hive.metastore.uris,执行hive --service metastore时会启动它 |
4. 安装MySQL
4.1. 安装MySQL
本文MySQL被安装在172.25.39.166机器上,Hive用MySQL来存储元数据,因此需要先安装好MySQL。这里安装的是最新的MySQL 5.6.17,下载网址为:http://dev.mysql.com/downloads/mysql/,本文选择是的“Linux - Generic”下的“Linux - Generic (glibc 2.5) (x86, 64-bit), Compressed TAR Archive”,它的二进制安装包名为mysql-5.6.17-linux-glibc2.5-x86_64.tar.gz。
将二进制安装包解压后,可看到名为INSTALL-BINARY的文件,该文件有说明如何安装MySQL,本文基本参照它进行的,MySQL安装目录为/data/hadoop/mysql,具体步骤如下(未使用mysql用户及mysql用户组,而是直接使用了当前登录用户hadoop,hadoop隶属用户组users):
# 以下均以当前用户hadoop执行 cd /data/hadoop tar xzf mysql-5.6.17-linux-glibc2.5-x86_64.tar.gz ln -s mysql-5.6.17-linux-glibc2.5-x86_64 mysql cd mysql scripts/mysql_install_db --user=hadoop bin/mysqld_safe --user=hadoop & cp support-files/mysql.server /etc/init.d/mysql.server # 这一条需求以root用户运行 |
4.2. 创建Hive元数据库
创建数据库hive:
create database if not exists hive; |
创建数据库用户hive:
create user hive identified by 'hive2014'; |
授权可以访问数据库hive的IP和用户,其中localhost的实际IP为172.25.39.166:
grant all on hive.* to 'hive'@'localhost' identified by 'hive2014'; grant all on hive.* to 'hive'@'172.25.39.166' identified by 'hive2014'; grant all on hive.* to 'hive'@'172.25.40.171' identified by 'hive2014'; |
进入hive数据库:
1) 本机进入:mysql -uhive -phive2014
2) 非本南进入:mysql -uhive -h172.25.39.166 -phive2014
5. 安装步骤
5.1. 下载Hive 0.12.0二进制安装包
下载网址:http://hive.apache.org/downloads.html,下载后的包名为:hive-0.12.0-bin.tar.gz,然后将hive-0.12.0-bin.tar.gz上传到/data目录下。
5.2. 安装Hive
1) 切换到/data目录:cd /data
2) 解压二进制安装包:tar xf hive-0.12.0-bin.tar.gz
3) 建立软链接:ln -s hive-0.12.0-bin hive
5.3. 安装MySQL-Connector
MySQL-Connector下载网址:http://dev.mysql.com/downloads/connector/。
选择“Connector/J”,接着选择“Platform Independent”,本文下载的是“mysql-connector-java-5.1.30.tar.gz”。
压缩包“mysql-connector-java-5.1.30.tar.gz”中有个mysql-connector-java-5.1.30-bin.jar,解压后将mysql-connector-java-5.1.30-bin.jar上传到Hive的lib目录下,这个是MySQL的JDBC驱动程序。
5.4. 修改配置
5.4.1. 修改/etc/profile或~/.profile
设置环境变量HIVE_HOME,并将Hive加入到PATH中:
export HIVE_HOME=/data/hadoop/hive export PATH=$HIVE_HOME/bin:$PATH |
5.4.2. 修改其它配置文件
进入/data/hadoop/hive/conf目录,可以看到如下:
hadoop@VM-40-171-sles10-64:~/hive/conf> ls hive-default.xml.template hive-exec-log4j.properties.template hive-env.sh.template hive-log4j.properties.template |
可以看到4个模板文件,复制并重命名成配置文件:
cp hive-env.sh.template hive-env.sh cp hive-default.xml.template hive-site.xml cp hive-log4j.properties.template hive-log4j.properties cp hive-exec-log4j.properties.template hive-exec-log4j.properties |
5.4.2.1. 修改hive-env.sh
如果之前没有设置好HADOOP_HOME环境变量,则可在hive-env.sh中,进行设置:
HADOOP_HOME=/data/hadoop/current |
5.4.2.2. 修改hive-site.xml
1) 修复BUG
该文件有个语法BUG,需要修改,进入到hive-site.xml的第2000行,该行内容为:<value>auth</auth>,明显的语法错误,需要将“auth”改成“value”。
2) 修改javax.jdo.option.ConnectionURL
将值设置为:jdbc:mysql://172.25.39.166:3306/hive?characterEncoding=UTF-8。
3) 修改javax.jdo.option.ConnectionDriverName
将值设置为:com.mysql.jdbc.Driver。
4) 修改javax.jdo.option.ConnectionUserName
将值设置为访问hive数据库的用户名hive:<value>hive</value>。
5) 修改javax.jdo.option.ConnectionPassword
将值设置为访问hive数据库的密码:<value>hive2014</value>。
6) 修改hive.metastore.schema.verification
该值试情况进行修改。
7) 修改hive.zookeeper.quorum
将值设置为:10.12.154.77,10.12.154.78,10.12.154.79,ZooKeeper被安装在这三台机器上。
8) 修改hive.metastore.uris
将值设置为:thrift://172.25.40.171:9083,9083为Hive元数据的RPC服务端口。
9) 修改hive.metastore.warehouse.dir
将值设置为:/data/hadoop/hive/warehouse,注意启动前,需要创建好该目录(mkdir /data/hadoop/hive/warehouse)。
10) 修改hive.server2.thrift.bind.host
该值默认为localhost,如果需要在其它机器远程访问Hive,则需要将它改成IP地址,本文将它改成172.25.40.171。
5.4.2.3. 修改hive-log4j.properties
修改日志文件存放目录,将日志目录由/tmp/${user.name}改为/data/hadoop/hive/logs:
hive.log.dir=/data/hadoop/hive/logs |
然后创建好目录/data/hadoop/hive/logs。
5.4.2.4. 修改hive-exec-log4j.properties
修改日志文件存放目录,将日志目录由默认的/tmp/${user.name}改为/data/hadoop/hive/logs/exec:
hive.log.dir=/data/hadoop/hive/logs/exec |
然后创建好目录/data/hadoop/hive/logs/exec。
6. 启动运行
1) 初始化metastore
安装配置好后,在启动Hive服务端之前,需要在服务端执行一次“schematool -dbType mysql -initSchema”,以完成对metastore的初始化。
2) 启动metastore
执行命令:hive --service metastore &
3) 启动Hive服务
执行:hiveserver2 &。
4) 进入Hive命令行操作界面(类似于mysql)
执行:hive
除了使用hive命令行操作界面之外,hiveserver2还提供了beeline(hive是用户名,hive2014是密码,可以从HiveServer2+Clients获得更多信息):
hadoop@VM-40-171-sles10-64:~/hive/bin> ./beeline Beeline version 0.12.0 by Apache Hive beeline> !connect jdbc:hive2://172.25.40.171:10000 hive hive2014 org.apache.hive.jdbc.HiveDriver Connecting to jdbc:hive2://172.25.40.171:10000 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/hadoop/hadoop-2.4.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/hadoop/hive-0.12.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Connected to: Hive (version 0.12.0) Driver: Hive (version 0.12.0) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://172.25.40.171:10000> select * from invites limit 2; +------+----------+-------+ | foo | bar | ds | +------+----------+-------+ | 474 | val_475 | 2014 | | 281 | val_282 | 2014 | +------+----------+-------+ 2 rows selected (1.779 seconds) 0: jdbc:hive2://172.25.40.171:10000> |
7. 远程执行HSQL
将hive/bin、hive/lib、hive/conf和hive/examples打包,如:tar czf hive-bin.tar.gz hive/bin hive/lib hive/conf hive/examples。
然后将hive-bin.tar.gz上传到其它机器,借助beeline即可远程执行HSQL(用hive可能会遇到问题,本文在操作时,使用hive,在执行HSQL时总会卡住,日志也没有记录特别原因,暂未去定位)。
8. 基本命令
以下内容来自官网(GettingStarted),注意命令不区分大小写:
CREATE TABLE pokes (foo INT, bar STRING); CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING); SHOW TABLES; SHOW TABLES '.*s'; DESCRIBE invites; DROP TABLE pokes; |
Hive的安装目录下有个examples子目录,存储了示例用到的数据文件等。测试往表invites中加载数据,将文件../examples/files/kv2.txt加载到表invites中:
LOAD DATA LOCAL INPATH '../examples/files/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds='2014'); |
可以通过“select * from invites;”来检验加载情况,或者执行“select count(1) from invites;”。
9. 常见错误
1) Failed to connect to the MetaStore Server
如果运行hiveserver2,遇到下列错误后,推荐打开DEBUG日志级别,以更查看更详细的信息,将日志配置文件hive-log4j.properties中的“hive.root.logger=WARN,DRFA”改成“hive.root.logger=DEBUG,WARN,DRFA”即可。
2014-04-23 06:00:04,169 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server... 2014-04-23 06:00:05,173 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server... 2014-04-23 06:00:06,177 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server... 2014-04-23 06:00:07,181 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server... 2014-04-23 06:00:08,185 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server... 2014-04-23 06:00:09,194 ERROR service.CompositeService (CompositeService.java:start(74)) - Error starting services HiveServer2 org.apache.hive.service.ServiceException: Unable to connect to MetaStore! at org.apache.hive.service.cli.CLIService.start(CLIService.java:85) at org.apache.hive.service.CompositeService.start(CompositeService.java:70) at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) |
修改后,再次运行hiveserver2,日志变详细了,猜测是metastore没有起来,可以通过执行“hive --service metastore”来启动metastore。
2014-04-23 06:04:27,053 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://172.25.40.171:9083 2014-04-23 06:04:27,085 WARN hive.metastore (HiveMetaStoreClient.java:open(288)) - Failed to connect to the MetaStore Server... org.apache.thrift.transport.TTransportException: java.net.ConnectException: 拒绝连接 at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:283) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:104) at org.apache.hive.service.cli.CLIService.start(CLIService.java:82) at org.apache.hive.service.CompositeService.start(CompositeService.java:70) at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) |
2) Version information not found in metastore
执行“./hive --service metastore”报下面这样的错误原因是未对metastore进行初始化,需要执行一次“schematool -dbType mysql -initSchema”。
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:5638) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:5622) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124) at com.sun.proxy.$Proxy2.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:403) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:4263) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:4197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) |