CentOS7 安装HBASE,HIVE

CentOS7 安装HBASE,HIVE

1. 前期准备
   Hadoop已经安装 2.7.7. mysql5.6已经安装
   版本配套：HBASE 2.0.5 Hive:2.3.4

2. 上传hbase和hive软件包至hadoop目录并解压
   tar -zxvf hbase-2.0.5-bin.tar.gz -C /home/hadoop/
   tar -zxvf apache-hive-2.3.4-bin.tar.gz -C /home/hadoop/
   tar -zxvf zookeeper-3.4.9.tar.gz -C /home/hadoop/
   cd
   mv hbase-2.0.5/ hbase
   mv apache-hive-2.3.4-bin/ hive
   mv zookeeper-3.4.9/ zookeeper

HBASE安装：
1.修改配置文件

cp
/home/hadoop/zookeeper/conf/zoo_sample.cfg /home/hadoop/zookeeper/conf/zoo.cfg
vi /home/hadoop/zookeeper/conf/zoo.cfg

vi log4j.properties

Define some default values that can be overridden by system properties

zookeeper.root.logger=INFO, CONSOLE
zookeeper.console.threshold=INFO
zookeeper.log.dir=/home/hadoop/zookeeper/logs

mkdir /home/hadoop/zookeeper/logs

vi /home/hadoop/hbase/conf/hbase-site.xml


hbase.rootdir
hdfs://hadoop:9000/hbase


hbase.cluster.distributed
true


hbase.zookeeper.quorum
hadoop:2181

vi /home/hadoop/hbase/conf/hbase-env.sh
export HBASE_MANAGES_ZK=false

3. 测试
   启动：
   zkServer.sh start
   start-hbase.sh

命令：
create ‘t1’,’f1’
put ‘t1’,’r1’,’f1:name’,’fw’
scan ‘t1’

停止：
stop-hbase.sh
zkServer.sh stop

OK!

HIVE安装：
1.修改配置文件

Mysql的配置文件修改为：
[mysql]
default-character-set=utf8
[mysqld]
skip-name-resolve
port = 3306
server_id = 1
log_bin = mysql-bin
socket=/var/lib/mysql/mysql.sock
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
max_connections=200
character-set-server=utf8
default-storage-engine=INNODB
lower_case_table_name=1
max_allowed_packet=16M
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
binlog_format=ROW
[client]
socket = /var/lib/mysql/mysql.sock
default-character-set=utf8

cp hive-env.sh.template hive-env.sh

vi hive-env.sh

Set HADOOP_HOME to point to a specific hadoop install directory

HADOOP_HOME=/home/hadoop/hadoop-2.7.7

Hive Configuration Directory can be controlled by:

export HIVE_CONF_DIR=/home/hadoop/hive/conf/

mv hive-log4j2.properties.template hive-log4j2.properties
property.hive.log.dir = /home/hadoop/hive/logs

mkdir -p /home/hadoop/hive/logs

2. 配置MySQL作为Metastore
   cp mysql-connector-java-5.1.40.jar /home/hadoop/hive/lib/
   cp /home/hadoop/hive/conf/hive-default.xml.template /home/hadoop/hive/conf/hive-site.xml

hive.exec.scratchdir /user/hive/tmp hive.scratch.dir.permission 733 hive.metastore.warehouse.dir /user/hive/warehouse javax.jdo.option.ConnectionURL jdbc:mysql://hadoop:3306/hive?createDatabaseIfNotExist=true javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName hive javax.jdo.option.ConnectionPassword hive hive.cli.print.header true hive.cli.print.current.db true

3. 创建mysql hive库和用户
   mysql> create database hive;
   Query OK, 1 row affected (0.00 sec)

mysql> use hive;
Database changed
mysql> create user ‘hive’@‘hadoop’ identified by ‘hive’;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on hive.* to ‘hive’@‘hadoop’ identified by ‘hive’;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> grant all on hive.* to ‘hive’@’%’ identified by ‘hive’;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> quit;
Bye
[root@hadoop ~]#

4.运行hive
4.1 初始化DB

首先需要在/etc/my.cnf上面新增一个配置
binlog_format=ROW

schematool -dbType mysql -initSchema

如果报错了，需要根据错误信息，找到对应的解决办法。

hive

截止目前，hive和HBASE都OK了。

HIVE测试：

准备文件:
[hadoop@hadoop test]$ file --mime-encoding student.txt
student.txt: utf-8
[hadoop@hadoop test]$ cat student.txt
95002,刘晨,女,19,IS
95017,王风娟,女,18,IS
95018,王一,女,19,IS
95013,冯伟,男,21,CS
95014,王小丽,女,19,CS
95019,邢小丽,女,19,IS
95020,赵钱,男,21,IS
95003,王敏,女,22,MA
95004,张立,男,19,IS
95012,孙花,女,20,CS
95010,孔小涛,男,19,CS
95005,刘刚,男,18,MA
95006,孙庆,男,23,CS
95007,易思玲,女,19,MA
95008,李娜,女,18,CS
95021,周二,男,17,MA
95022,郑明,男,20,MA
95001,李勇,男,20,CS
95011,包小柏,男,18,MA
95009,梦圆圆,女,18,MA
95015,王君,男,18,MA

load data local inpath ‘/home/hadoop/test/student.txt’ overwrite into table student;

但是加载后发现中文乱码，哪怕用格式转换也不行。
iconv -f iso-8859-1 -t utf-8 student.txt -o student.txt.utf8

后来莫名其妙就OK了，其实就是终端字符集设置为UTF-8即可。