完全分布式安装Hadoop,Hive,Hbase,Hwi,Zookeeper-500行说明

最新推荐文章于 2024-05-28 10:55:52 发布

徐长亮

最新推荐文章于 2024-05-28 10:55:52 发布

阅读量3.5k

点赞数 3

分类专栏： [2016年以后] 基于hadoop的数据平台文章标签： hadoop安装 hive安装 hbase安装 zookeeper安装

本文链接：https://blog.csdn.net/clark_xu/article/details/69668618

版权

[2016年以后] 基于hadoop的数据平台专栏收录该内容

3 篇文章 0 订阅

订阅专栏

其他部分见：

HDFS YARN zookeeper HBASE HIVE HIVE hwi的启动

HDFS和HBASE动态增加和减少节点

一.下载介质到/opt下面：

1.hadoop
wget http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

2.下载jdk1.8.0_121
3.下载mysql
wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-community-server-5.7.17-1.el6.x86_64.rpm --no-check-certificate

4.下载hive
wget http://mirrors.cnnic.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
wget http://mirrors.cnnic.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-src.tar.gz

5.下载zookeeper
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.2-alpha/zookeeper-3.5.2-alpha.tar.gz   --no-check-certificate

6.下载hbase
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/1.3.0/hbase-1.3.0-bin.tar.gz --no-check-certificate

7.下载其他：
#下载storm
[root@node02 opt]# wget https://mirrors.cnnic.cn/apache/storm/apache-storm-1.1.0/apache-storm-1.1.0.tar.gz --no-check-certificate
#下载sqoop
wget https://mirrors.cnnic.cn/apache/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz --no-check-certificate
#下载spark
wget https://mirrors.cnnic.cn/apache/spark/spark-2.1.0/spark-2.1.0-bin-hadoop2.7.tgz --no-check-certificate
wget https://mirrors.cnnic.cn/apache/spark/spark-2.1.0/spark-2.1.0-bin-without-hadoop.tgz --no-check-certificate
#下载kafka
https://mirrors.cnnic.cn/apache/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz --no-check-certificate
#下载flume
wget https://mirrors.cnnic.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz --no-check-certificate

二.准备四个节点：

1.node01,node02,node03,node04,而且已经配置了ssh互信，而且已经关闭了防火墙
2.应用解压安装目录为/app
3.HDFS目录为：/app/dirhadoop/

三.安装hadoop

1.环境变量---四个节点都要执行

echo 'export JAVA_HOME=/app/jdk1.8.0_121' >> /etc/profile
echo 'export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar' >> /etc/profile
echo 'export HADOOP_HOME=/app/hadoop-2.7.3' >>/etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin' >> /etc/profile
source /etc/profile

2.创建HDFS使用的目录：--四个节点都要执行

#core-site.xml的hadoop.tmp.dir
mkdir -p /app/dirhadoop/tmp
#hdfs-site.xml的dfs.namenode.name.dir
mkdir -p /app/dirhadoop/dfs/name
#hdfs-site.xml的dfs.datanode.data.dir
mkdir -p /app/dirhadoop/dfs/data

3.配置hadoop-env.sh

echo 'export JAVA_HOME=/app/jdk1.8.0_121'  >>/app/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

4.配置core-site.xml

(1)hadoop.tmp.dir定义了hdfs的临时文件存放目录
(2)>fs.default.name指定了如何访问HDFS，例如hbase设置为：
    <name>hbase.rootdir</name><value>hdfs://node01:9000/hbase</value> 


#vi /app/hadoop-2.7.3/etc/hadoop/core-site.xml
<configuration>  
  <property>  
    <name>fs.default.name</name>  
    <value>hdfs://node01:9000</value>  
  </property>  
  <property>  
    <name>hadoop.tmp.dir</name>  
    <value>/app/dirhadoop/tmp</value>  
  </property>  
</configuration>

5.配置hdfs-site.xml

(1)dfs.namenode.name.dir
(2)dfs.datanode.data.dir

<configuration>  
  <property>  
    <name>dfs.replication</name>  
    <value>2</value>  
  </property>  
  <property>  
    <name>dfs.namenode.name.dir</name>  
    <value>file:/app/dirhadoop/dfs/name</value>  
  </property>  
  <property>  
    <name>dfs.datanode.data.dir</name>  
    <value>file:/app/dirhadoop/dfs/data</value>  
  </property>  
</configuration>

6.配置mapred-site.xml

(1)mapreduce.jobhistory.webapp.address
定义了访问网址
(2)部署完成之后，可以通过sbin/mr-jobhistory-daemon.sh start historyserver
启动mapreduce.jobhistory.webapp


#vi /app/hadoop-2.7.3/etc/hadoop/mapred-site.xml

<configuration>  
  <property>  
    <name>mapreduce.framework.name</name>  
    <value>yarn</value>  
  </property>  
  <property>  
    <name>mapreduce.jobhistory.address</name>  
    <value>node01:10020</value>  
  </property>  
  <property>  
    <name>mapreduce.jobhistory.webapp.address</name>  
    <value>node01:19888</value>  
  </property>  
</configuration>

7.配置yarn-site.xml

(1)
#vi /app/hadoop-2.7.3/etc/hadoop/slaves
node02
node03
node04

(2)
a.yarn.resourcemanager.webapp.address定义网页访问：
/app/hadoop-2.7.3/sbin/start-yarn.sh启动之后，就可以类似
http://192.168.13.129:8088/cluster访问了

#vi /app/hadoop-2.7.3/etc/hadoop/yarn-site.xml

<configuration>   
<!-- Site specific YARN configuration properties -->  
  <property>  
    <name>yarn.nodemanager.aux-services</name>  
    <value>mapreduce_shuffle</value>  
  </property>  
  <property>  
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.address</name>  
    <value>node01:8032</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.scheduler.address</name>  
    <value>node01:8030</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.resource-tracker.address</name>  
    <value>node01:8031</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.admin.address</name>  
    <value>node01:8033</value>  
  </property>  
  <property>  
    <name>yarn.resourcemanager.webapp.address</name>  
    <value>node01:8088</value>  
  </property>  
</configuration>

8.node01完成以上配置之后：

scp -r /app/hadoop-2.7.3 node02:/app/
scp -r /app/hadoop-2.7.3 node03:/app/
scp -r /app/hadoop-2.7.3 node04:/app/

9.在master节点(node01)格式化hdfs

hdfs namenode -format

10.完成之后，master节点启动HDFS

(1)启动hdfs:
[root@node01 ~]# /app/hadoop-2.7.3/sbin/start-dfs.sh
Starting namenodes on [node01]
node01: Warning: Permanently added the RSA host key for IP address '192.168.13.129' to the list of known hosts.
node01: starting namenode, logging to /app/hadoop-2.7.3/logs/hadoop-root-namenode-node01.out
node02: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node02.out
node03: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node03.out
node04: starting datanode, logging to /app/hadoop-2.7.3/logs/hadoop-root-datanode-node04.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /app/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-node01.out

网页查看：
http://192.168.13.129:50070/dfshealth.html#tab-overview

(2)启动yarn
[root@node01 ~]# /app/hadoop-2.7.3/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-resourcemanager-node01.out
node03: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node03.out
node02: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node02.out
node04: starting nodemanager, logging to /app/hadoop-2.7.3/logs/yarn-root-nodemanager-node04.out

网页查看：<name>yarn.resourcemanager.webapp.address</name>  
 
http://192.168.13.129:8088/cluster/nodes

(1)启动historyserver
sbin/mr-jobhistory-daemon.sh start historyserver
http://192.168.13.129:19888/jobhistory

四.安装Zookeeper

1.单个创建文件myid

hosts为node01,node02,node03,node04,也可以是IP

[root@node01 zookeeper-3.5.2-alpha]# cat myid
node01	1
node02	2
node03	3
node04	4

2.单个节点配置zoo.cfg

#[root@node01 conf]# cat zoo_sample.cfg
#vi zoo.cfg
tickTime=2000
dataDir=/app/zookeeper-3.5.2-alpha
clientPort=2181
initLimit=5
syncLimit=2
server.1=node01:2888:3888 
server.2=node02:2888:3888
server.3=node03:2888:3888
server.4=node04:2888:3888

#server.1,server.2,server.3,server.4和myid相匹配

3.4 复制和启动

3.Zookeeper复制到其他所有节点：
scp -r /app/zookeeper-3.5.2-alpha node02：/app
scp -r /app/zookeeper-3.5.2-alpha node03：/app
scp -r /app/zookeeper-3.5.2-alpha node04：/app

4.每个节点执行：
/app/zookeeper-3.5.2-alpha/bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /app/zookeeper-3.5.2-alpha/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

(1)状态：/app/zookeeper-3.5.2-alpha/bin/zkServer.sh status
Mode: leader
Mode: follower

5.报错处理

[root@node01 logs]# cat zookeeper-root-server-node01.out 
2017-04-07 15:47:22,297 [myid:] - INFO  [main:QuorumPeerConfig@116] - Reading configuration from: /app/zookeeper-3.5.2-alpha/bin/../conf/zoo.cfg
2017-04-07 15:47:22,304 [myid:] - INFO  [main:QuorumPeerConfig@318] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2017-04-07 15:47:22,304 [myid:] - INFO  [main:QuorumPeerConfig@322] - secureClientPort is not set
2017-04-07 15:47:22,311 [myid:] - ERROR [main:QuorumPeerMain@86] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Address unresolved: node01:3888 
	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.<init>(QuorumPeer.java:242)
	at org.apache.zookeeper.server.quorum.flexible.QuorumMaj.<init>(QuorumMaj.java:89)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.createQuorumVerifier(QuorumPeerConfig.java:524)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseDynamicConfig(QuorumPeerConfig.java:557)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:530)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:353)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:133)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:110)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

分析：这个报错原因是zoo.cfg配置文件，server.1=node01:2888:3888后面有空格导致

五.安装HBASE

1.每个节点执行环境变量：

echo 'export HBASE_HOME=/app/hbase-1.3.0' >> /etc/profile
echo 'export PATH=$PATH:$HBASE_HOME/bin' >> /etc/profile
source /etc/profile

2.配置hbase-env.sh

(1)
mkdir logs #Hbase日志目录

(2)
echo 'export JAVA_HOME=/app/jdk1.8.0_121' >> conf/hbase-env.sh
echo 'export HADOOP_HOME=/app/hadoop-2.7.3'>> conf/hbase-env.sh
echo 'export HBASE_HOME=/app/hbase-1.3.0' >> conf/hbase-env.sh
echo 'export HBASE_MANAGES_ZK=false ' >> conf/hbase-env.sh
echo 'export HBASE_LOG_DIR=/app/hbase-1.3.0/logs' >> conf/hbase-env.sh

3.配置hbase-site.xml

(1)hbase.master.info.port指定了hbase web客户端的访问网址
(2)hbase.rootdir指定了在HDFS中的位置，根目录参考core-sit.xml的fs.default.name
(3)hbase.zookeeper相关属性参照zookeeper的配置：

<configuration>  
  <property>  
    <name>hbase.rootdir</name>  
    <value>hdfs://node01:9000/hbase</value>  
  </property>  
  <property>  
     <name>hbase.cluster.distributed</name>  
     <value>true</value>  
  </property>  
  <property>  
      <name>hbase.master</name>  
      <value>node01:60000</value>  
  </property>  
   <property>  
    <name>hbase.zookeeper.property.dataDir</name>  
    <value>/app/zookeeper-3.5.2-alpha</value>  
  </property>  
  <property>  
    <name>hbase.zookeeper.quorum</name>  
    <value>node01,node02,node03,node04</value>  
  </property>  
  <property>  
    <name>hbase.zookeeper.property.clientPort</name>  
    <value>2181</value>  
  </property> 
  <property>
  <name>hbase.master.info.port</name>
  <value>60010</value>
</property> 
</configuration>

4.配置regionservers

[root@node01 hbase-1.3.0]# cat conf/regionservers
node02
node03
node04

5.scp到各个节点:

6.启动

  启动之前得保证ZK和hadoop已经启动
  [master@master1 hbase]$ bin/start-hbase.sh

7.报错处理：

(1)hbase(main):006:0> t1 = create 't1', 'f1'
ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
(2)是四个节点时间不同步
处理：
[root@node01 ~]# cat /etc/ntp.conf
restrict default nomodify
restrict 127.0.0.1
server 127.127.1.0
fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
broadcastdelay 0.008

[root@node02 ~]# ntpdate node01
 7 Apr 16:51:55 ntpdate[14684]: step time server 192.168.85.129 offset -38.280736 sec
[root@node03 ~]# ntpdate node01
 7 Apr 16:52:04 ntpdate[9019]: step time server 192.168.85.129 offset -40.609528 sec
[root@node04 ~]# ntpdate node01
 7 Apr 16:52:14 ntpdate[9042]: step time server 192.168.85.129 offset -38.668817 sec

六.安装HIVE

1.安装mysql

(1)
yum install -y mysql-server mysql mysql-deve
groupadd mysql
useradd -g mysql mysql
chown -R mysql:mysql /var/lib/mysql

(2)
[root@node01 ~]# cat /etc/my.cnf 
[mysqld]
datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

(3)
[root@node01 ~]# service mysqld start
Starting mysqld:                                           [  OK  ]

2.配置mysql

#设置密码
mysqladmin -u root password 'root'
#赋给权限
create user 'hive' identified by 'hive'
grant all privileges on *.* to 'hive' with grant option;
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'node01' IDENTIFIED BY 'hive' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
flush privileges;
#测试
mysql -h node01 -u hive -p

3.每个节点执行环境变量

echo 'export HIVE_HOME=/app/apache-hive-2.1.1-bin' >> /etc/profile
echo 'export PATH=$PATH:$HIVE_HOME/bin' >> /etc/profile
source /etc/profile

4.修改hive-env.sh

cp /app/apache-hive-2.1.1-bin/conf/hive-env.sh.template /app/apache-hive-2.1.1-bin/conf/hive-env.sh
/**需要添加**/
echo 'export JAVA_HOME=/app/jdk1.8.0_121'  >>/app/apache-hive-2.1.1-bin/conf/hive-env.sh
ln -s $JAVA_HOME/lib/tools.jar $HIVE_HOME/lib/

5.放置mysql驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
cp mysql-connector-java-5.1.41-bin.jar /app/apache-hive-2.1.1-bin/lib/

6.修改hive-site.xml

1)javax.jdo.option.ConnectionURL安装mysql的节点
(2)javax.jdo.option.ConnectionUserName mysql的元数据的用户
(3)javax.jdo.option.ConnectionPassword密码

#cp hive-default.xml.template hive-site.xml
<configuration>
<property>  
  <name>javax.jdo.option.ConnectionURL</name>  
  <value>jdbc:mysql://node01:3306/hive?createDatabaseIfNotExist=true</value>  
  <description>JDBC connect string for a JDBC metastore</description>  
</property>  
  
<property>  
  <name>javax.jdo.option.ConnectionDriverName</name>  
  <value>com.mysql.jdbc.Driver</value>  
  <description>Driver class name for a JDBC metastore</description>  
</property>  
  
<property>  
  <name>javax.jdo.option.ConnectionUserName</name>  
  <value>hive</value>  
  <description>username to use against metastore database</description>  
</property>  
  
<property>  
  <name>javax.jdo.option.ConnectionPassword</name>  
  <value>hive</value>  
  <description>password to use against metastore database</description>  
</property>  
</configuration>

7.复制hive到其他节点

scp -r /app/apache-hive-2.1.1-bin/  node02:/app/
scp -r /app/apache-hive-2.1.1-bin/  node03:/app/
scp -r /app/apache-hive-2.1.1-bin/  node04:/app/

8.创建HDFS目录

hadoop fs -mkdir /hive
hadoop fs -mkdir /hive/warehouse
hadoop fs -chmod g+w /hive
hadoop fs -chmod g+w /hive/warehouse

9.初始化元数据

schematool -initSchema -dbType mysql

10.建表测试

create table test_ds  
(  
  id int comment '用户ID',  
  name string comment '用户名称'  
)  
comment '测试分区表'  
partitioned by(ds string comment '时间分区字段')  
clustered by(id) sorted by(name) into 32 buckets      
row format delimited   
fields terminated by '\t'  
stored as rcfile;

七.安装hive hwi

1.打包hive-hwi-2.1.1.war

[root@node01 hwi]# pwd
/app/hive_src/apache-hive-2.1.1-src
jar cvfM0 hive-hwi-2.1.1.war -C web/ .
cp hive-hwi-2.1.1.war /app/apache-hive-2.1.1-bin/lib/

2.hive-site.xml中添加

<property>
    <name>hive.hwi.war.file</name>
    <value>lib/hive-hwi-2.1.1.war</value>
    <description>This sets the path to the HWI war file, relative to ${HIVE_HOME}. </description>
</property>
<property>
    <name>hive.hwi.listen.host</name>
    <value>node01</value>
    <description>This is the host address the Hive Web Interface will listen on</description>
</property>
<property>
    <name>hive.hwi.listen.port</name>
    <value>9999</value>
    <description>This is the port the Hive Web Interface will listen on</description>
</property>

3.启动和查看

[root@node01 lib]# pwd
/app/apache-hive-2.1.1-bin/lib
[root@node01 lib]# nohup hive --service hwi &

查看
http://192.168.85.129:9999/hwi/index.jsp

支持原创

公众号：clark_blog
博客：blog.csdn.NET/clark_xu

代码：https://github.com/clark99

徐长亮

关注

3
点赞
踩
4

收藏

觉得还不错? 一键收藏
1
评论
完全分布式安装Hadoop,Hive,Hbase,Hwi,Zookeeper-500行说明

HDFS YARN zookeeper HBASE HIVE HIVE hwi的启动HDFS和HBASE动态增加和减少节点完全分布式安装Hadoop,Hive,Hbase,Hwi,Zookeeper
复制链接

扫一扫