Linux操作系统hadoop+zookeeper+hbase 集群部署
1.1.安装包
1)准备至少3台PC或虚拟机
2)Linux系统:CentOS-7.0-1406-x86_64-DVD.iso
3)Java安装包:jdk-8u161-linux-x64.tar.gz
4)Hadoop安装包:hadoop-2.8.5.tar.gz
5)Hbase安装包:hbase-2.2.5-bin.tar.gz
1.2.网络配置
主机名 | IP |
master | 192.168.1.53 |
slave1 | 192.168.1.54 |
slave2 | 192.168.1.55 |
二.安装配置CentOS
2.1安装CentOS
1)选择启动盘CentOS-7.0-1406-x86_64-DVD.iso,启动安装
2)选择Install CentOS 7,回车,继续安装
3)选择语言,默认是English,学习可以选择中文,正式环境选择English
4)配置网络和主机名,主机名(hostname):master,网络选择开启,配置手动的IPV4
修改hostname:vi /proc/sys/kernel/hostname
5)选择安装位置;在分区处选择手动配置;选择标准分区,点击这里自动创建他们,点击完成,收受更改
6)修改root密码,密码:
7)重启,安装完毕。
2.2配置IP
2.2.1检查IP
# ip addr或# ip link
2.2.2配置IP和网管
#cd /etc/sysconfig/network-scripts #进入网络配置文件目录
# find ifcfg-em* #查到网卡配置文件,例如ifcfg-em1
# vi ifcfg-em1 #编辑网卡配置文件或
# vi /etc/sysconfig/network-scripts/ifcfg-em1 #编辑网卡配置文件
配置内容:
BOOTPROTO=static #静态IP配置为static,动态配置为dhcp
ONBOOT=yes #开机启动
IPADDR=192.168.1.53 #IP地址
NETMASK=255.255.255.0 #子网掩码
GATEWAY=192.168.1.1
DNS1=144.144.144.144
# systemctl restart network.service #重启网络
2.2.3配置hosts
# vi /etc/hosts
删除或注释其他不需要的配置
新增:
192.168.1.53 master
192.168.1.54 slave1
192.168.1.55 slave2
2.3关闭防火墙
# systemctl status firewalld.service #检查防火墙状态
# systemctl stop firewalld.service #关闭防火墙
# systemctl disable firewalld.service #禁止开机启动防火墙
2.4时间同步
# yum install -y ntp #安装ntp服务
# ntpdate cn.pool.ntp.org #同步网络时间
2.5安装配置jdk
2.5.1卸载自带jdk
用命令java –version查看是否系统已经安装jdk,如果已经安装且版本不适用,需要先卸载:
先查看rpm -qa | grep java
显示如下信息:
java-1.4.2-gcj-compat-1.4.2.0-40jpp.115
java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
卸载:
rpm -e--nodepsjava-1.4.2-gcj-compat-1.4.2.0-40jpp.115
rpm -e --nodepsjava-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
还有一些其他的命令
rpm-qa | grep gcj
rpm-qa | grep jdk
如果出现找不到openjdksource的话,那么还可以这样卸载
yum -yremove javajava-1.4.2-gcj-compat-1.4.2.0-40jpp.115
yum -y remove javajava-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
2.5.2安装jdk
上传jdk-8u161-linux-x64.gz 安装包到root根目录
# mkdir /opt
# tar -zxvf jdk-8u161-linux-x64.gz -C /opt/
# rm –rf jdk-8u161-linux-x64.gz
2.5.3各个主机之间复制jdk
# scp -r /opt slave1:/usr
# scp -r /opt slave2:/usr
2.5.4各个主机配置jdk环境变量
# vi /etc/profile
编辑内容
export JAVA_HOME=/opt/jdk1.8.0_161
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
# source /etc/profile #使配置文件生效
# java -version #查看java版本
2.6配置ssh无密钥访问
分别在各个主机上检查ssh服务状态:
# systemctl status sshd.service #检查ssh服务状态
# yum install openssh-server openssh-clients #安装ssh服务,如果已安装,则不用执行该步骤
# systemctl start sshd.service #启动ssh服务,如果已安装,则不用执行该步骤
分别在各个主机上生成密钥
# ssh-keygen -t rsa #生成密钥
在slave1上
# cp ~/.ssh/id_rsa.pub ~/.ssh/slave1.id_rsa.pub
#scp ~/.ssh/slave1.id_rsa.pub master:~/.ssh
在slave2上
# cp ~/.ssh/id_rsa.pub ~/.ssh/slave2.id_rsa.pub
# scp ~/.ssh/slave2.id_rsa.pub master:~/.ssh
在master上
# cd ~/.ssh
# cat id_rsa.pub >> authorized_keys
# cat slave1.id_rsa.pub >>authorized_keys
# cat slave2.id_rsa.pub >>authorized_keys
# scp authorized_keys slave1:~/.ssh
# scp authorized_keys slave2:~/.ssh
三.安装配置hadoop
3.1安装hadoop
上传hadoop-2.8.5.tar.gz安装包到root根目录,以下目录可以不用再此处新建,hdfs在format的时候会自己创建。
# tar -zxvf hadoop-2.8.5.tar.gz -C /usr
# rm -rf hadoop-2.8.5.tar.gz
# mkdir /opt/hadoop/tmp
# mkdir /opt/hadoop/logs
# mkdir /opt/hadoop/hdf
# mkdir /opt/hadoop/hdf/data
# mkdir /opt/hadoop/hdf/name
3.1.1在hadoop中配置hadoop-env.sh文件
# vi /opt/hadoop/etc/hadoop/hadoop-env.sh
修改java_home:
export JAVA_HOME=/opt/jdk
3.1.2修改yarn-env.sh
# vi /opt/hadoop/etc/hadoop/yarn-env.sh
exportJAVA_HOME=/opt/jdk
3.1.3修改slaves
# vi /opt/hadoop/etc/hadoop/slaves
配置内容:
删除:localhost
添加:
slave1
slave2
3.1.4修改core-site.xml
# vi /opt/hadoop/etc/hadoop/core-site.xml
配置内容:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop/tmp</value>
</property>
</configuration>
3.1.5修改hdfs-site.xml
# vi /opt/hadoop/etc/hadoop/hdfs-site.xml
配置内容:
<configuration>
<property>
<name>dfs.datanode.data.dir</name>
<value>/opt/hadoop/hdf/data</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/hdf/name</value>
<final>true</final>
</property>
</configuration>
3.1.6修改mapred-site.xml
# cp /opt/hadoop/etc/hadoop/mapred-site.xml.template /opt/hadoop/etc/hadoop/mapred-site.xml
# vi /opt/hadoop/etc/hadoop/mapred-site.xml
配置内容:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
3.1.7修改yarn-site.xml
# vi /opt/hadoop/etc/hadoop/yarn-site.xml
配置内容:
<configuration>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
3.2各个主机之间复制hadoop
# scp -r /opt/hadoop slave1:/opt
# scp -r /opt/hadoop slave2:/opt
3.3各个主机配置hadoop环境变量
# vi /etc/profile
编辑内容:
export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_LOG_DIR=/opt/hadoop/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
# source /etc/profile #使配置文件生效
3.4格式化namenode
# cd /opt/hadoop/sbin
# hdfs namenode -format
3.5启动hadoop
启动hdfs:
# cd /opt/hadoop/sbin
# start-all.sh
检查hadoop启动情况:
http://192.168.1.53:50070
http://192.168.1.53:8088/cluster
检查进程:
# jps
master主机包含ResourceManager、SecondaryNameNode、NameNode等,则表示启动成功,例如
2212 ResourceManager
2484 Jps
1917 NameNode
2078 SecondaryNameNode
各个slave主机包含DataNode、NodeManager等,则表示启用成功,例如
17153 DataNode
17334 Jps
17241 NodeManager
四.安装配置zookeeper
4.1配置zookeeper环境变量
#vi /etc/profile
export ZOOKEEPER_HOME=/opt/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH
#source /etc/profile
4.2配置zookeeper
1、到zookeeper官网下载:
http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/
2、搭建zookeeper
所有主机都要配置
3、上传zookeeper-3.4.6.tar.gz到master服务器的根目录,并解压:zookeeper:
tar –zxvf zookeeper-3.4.6.tar.gz -C /usr
4、在zookeeper目录下建立zookeeper-data目录,同时将zookeeper目录下conf/zoo_simple.cfg文件复制一份成zoo.cfg
cp /usr/zookeeper-3.4.6/conf/zoo_sample.cfg zoo.cfg
5、修改zoo.cfg
# Thenumber of milliseconds of each tick
tickTime=2000
# Thenumber of ticks that the initial
#synchronization phase can take
initLimit=10
# Thenumber of ticks that can passbetween
#sending a request and getting anacknowledgement
syncLimit=5
# thedirectory where the snapshot isstored.
# do notuse /tmp for storage, /tmp hereis just
#example sakes.
dataDir=/usr/zookeeper-3.4.6/zookeeper-data
# theport at which the clients willconnect
clientPort=2181
# themaximum number of clientconnections.
#increase this if you need to handle moreclients
#maxClientCnxns=60
# Besure to read the maintenance sectionof the
# administratorguide before turning onautopurge.
#http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
# Thenumber of snapshots to retain indataDir
#autopurge.snapRetainCount=3
# Purgetask interval in hours
# Set to"0" to disable autopurge feature
#autopurge.purgeInterval=1
server.0=master:2888:3888
server.1=slave1:2888:3888
server.2=slave2:2888:3888
6、拷贝zookeeper目录到另外两台服务器:
scp –r /usr/zookeeper-3.4.6 slave1:/usr
scp –r /usr/zookeeper-3.4.6 slave2:/usr
分别在几台服务器的zookeeper-data目录下建立myid其ip对应相应的server.*:master的myid内容为0;slave1的myid内容为1;slave2的myid为2。例如master的mydi文件创建:
cd /usr/zookeeper-3.4.6/zookeeper-data
vi myid
编辑内容:
0
保存退出即可。
7、启动ZooKeeper集群,在每个节点上分别启动ZooKeeper服务:
cd /opt/zookeeper/bin
./zkServer.sh start
8、查看ZooKeeper集群的状态:
cd /opt/zookeeper/
./zkServer.sh status
五.安装配置hbase
5.1安装hbase
上传hbase-2.2.5-bin.tar.gz安装包到root根目录
# tar -zxvf hbase-2.2.5-bin.tar.gz -C /usr
# mkdir /usr/hbase-2.2.5/logs
5.2配置hbase环境变量
vi /etc/profile
增加配置:
export HBASE_HOME=/usr/hbase
export PATH=$PATH:$HBASE_HOME/bin
保存退出,并使其生效:
source /etc/profile
5.3修改hbase-env.sh
# vi /usr/hbase-2.2.5/conf/hbase-env.sh
配置内容:
export JAVA_HOME=/opt/jdk1.8.0_161
export HBASE_LOG_DIR=${HBASE_HOME}/logs
export HBASE_MANAGES_ZK=false
5.4修改regionservers
# vi /usr/hbase-2.2.5/conf/regionservers
配置内容:
删除:localhost
添加:
slave1
slave2
5.5修改hbase-site.xml
# vi /usr/hbase-2.2.5/conf/hbase-site.xml
配置内容:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,slave1,slave2</value>
</property>
<property>
<name>hbase.master</name>
<value>hdfs://master:60000</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/zookeeper-3.4.6/zookeeper-data</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
</configuration>
5.6各个主机之间复制hbase
# scp -r /usr/hbase-2.2.5 slave1:/usr
# scp -r /usr/hbase-2.2.5 slave2:/usr
5.7启动hbase
启动之前先启动hadoop和zookeeper集群
启动hbase:
# cd /usr/hbase-2.2.5/bin
#./start-hbase.sh
5.8查看hbase状态
# hbase shell
#hbase(main):011:0> status
1 active master, 0 backup masters, 2 servers, 0dead, 4.0000 average load
5.8创建表
六.集群启动与停止
6.1启动集群
6.1.1启动每台服务器的zookeeper服务
# cd /usr/zookeeper-3.4.6/bin
# ./zkServer.sh start
6.1.2启动master服务器的hadoop服务
# cd /opt/hadoop/sbin
# start-all.sh
6.1.3启动master服务器的hbase服务
# cd /usr/hbase-2.2.5/bin
#./start-hbase.sh
6.2停止集群
6.1.1停止master服务器的hbase服务
# cd /usr/hbase-2.2.5/bin
#./stop-hbase.sh
6.1.2停止master服务器的hadoop服务
# cd /opt/hadoop/sbin
# stop-all.sh
6.1.3停止每台服务器的zookeeper服务
# cd /usr/zookeeper-3.4.6/bin
# ./zkServer.sh stop
6.1.4强制停止
如无法正常停止,可使用kill命令杀掉进程。