文章目录
一、Zookeeper入门教程
1 Zookeeper定义
Apache Zookeeper是一个开发和维护开源服务器得框架,它可以实现高度可靠的分布式协调。
2 Zookeeper的特点
- 半数以上存活即存活
- 搭建奇数台节点
- 生产中没有使用单台Zookeeper的情况,也就是至少3台
- 集群中只有一个leader,多个follower
- 全局数据保持一致,每个server保存一份相同的数据副本,client无论连接到哪一个server,数据都保持一致
- 分布式读写,更新请求转发,由leader实施(主从复制,读写分离)
- 更新请求顺序进行,来自同一个client的更新请求按照其发送的请求顺序执行(这个特点决定ZK不适合做海量数据的保存)
- 数据更新原子性,一次数据更新操作,要么全部成功,要么全部失败
- 实时性,在一定时间范围内,client能读到最新数据
3 Zookeeper安装
##1.解压缩
tar -zxvf zookeeper-3.4.5.tar.gz -C /opt/apps/
##2.配置环境变量
vim /etc/profile
export ZK_HOME=/opt/apps/zookeeper-3.4.5
export PATH=$PATH:$ZK_HOME/bin
source /etc/profile
##3.创建zkdata/myid,每个节点myid不同
eg1:192.168.130.110
mkdir zkdata
vim zkdata/myid
1
eg2:192.168.130.111
mkdir zkdata
vim zkdata/myid
2
eg2:192.168.130.112
mkdir zkdata
vim zkdata/myid
3
##4.配置zoo.cfg
mv conf/zoo_sample.cfg conf/zoo.cfg
vim zoo.cfg
dataDir=/opt/apps/zookeeper-3.4.5/zkdata
server.1=192.168.130.110:2888:3888
server.2=192.168.130.111:2888:3888
server.3=192.168.130.112:2888:3888
##5.启动
##服务端
sh zkServer.sh start/stop/status
##客户端
sh zkCli.sh
4 Zookeeper操作
指令 | 作用 | 案例 |
---|---|---|
ls | 列举指定目录的子目录 | ls / |
ls2 | 列举指定目录的子目录以及元数据 | ls2 / |
create | 创建新节点 -s 顺序节点 -e 临时节点 -c 容器节点 -t 可以给节点添加过期时间,默认禁用, 可以通过系统参数启用 | create /hdfs hdfs create -e /yarn yarn create -s /mapreduce mr |
get | 获取指定节点中的数据以及元数据 | get /hdfs |
stat | 获取节点的元数据 | stat /hdfs |
delete | 删除指定节点-要求它没有子节点 | delete /hdfs |
rmr | 递归删除znode | rmr /hdfs |
set | 修改指定节点的数据 | set /hdfs hadoop |
5 Java API
5.1 导入Pom依赖
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.6</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
</dependency>
5.2 配置本地的hosts
C:\Windows\System32\drivers\etc目录下的hosts文件,映射对应的IP
192.168.130.110 ha1
192.168.130.111 ha2
192.168.130.112 ha3
5.3 代码
- Operation.java
package com.zookeeper;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.junit.Test;
public class Operation {
private static ZooKeeper zooKeeper = null;
@Test
public void init(){
String connection = "ha1,ha2,ha3";
int TimeOut = 3000;
try {
//操作zookeeper的服务器客户端对象
zooKeeper = new ZooKeeper(connection, TimeOut, new Watcher() {
@Override
public void process(WatchedEvent watchedEvent) {
System.out.println("zookeeper");
}
});
CURD.setZooKeeper(zooKeeper);
/**
* JavaAPI模拟指令
*/
CURD curd = new CURD();
//关闭连接
zooKeeper.close();
}catch (Exception e){
e.printStackTrace();
}
//释放资源
}
}
- CURD.java
package com.zookeeper;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.Test;
import java.util.List;
public class CURD {
private static ZooKeeper zooKeeper;
public static ZooKeeper getZooKeeper() {
return zooKeeper;
}
public static void setZooKeeper(ZooKeeper zooKeeper) {
CURD.zooKeeper = zooKeeper;
}
//查看路径下所有子目录
@Test
public void lsAll(){
try {
List<String> children = zooKeeper.getChildren("/", true);
for (String child:children){
System.out.println(child);
}
}catch (Exception e){
e.printStackTrace();
}
}
//创建节点
@Test
public void create(){
try {
zooKeeper.create("/hdfs","hdfs".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
}catch (Exception e){
e.printStackTrace();
}
}
//删除节点
@Test
public void delete(){
try {
zooKeeper.delete("/hadoop",1);
}catch (Exception e){
e.printStackTrace();
}
}
//递归删除节点
@Test
public void rmr(){
rmrs("/hdfs");
}
@Test
public void rmrs(String path){
try {
String origin = path;
//先获取子节点,判断有没有子节点
List<String> children = zooKeeper.getChildren(path, true);
//没有节点就删除
if (children.size() == 0){
zooKeeper.delete(path,0);
return;
}
//有节点就删除
for (String child : children){
path = path + "/" + child;
rmrs(path);
}
//删除自己本身节点
zooKeeper.delete(origin,0);
}catch (Exception e){
e.printStackTrace();
}
}
//获取指定节点中的数据以及元数据
@Test
public void get() {
try {
Stat stat = new Stat();//元数据
byte[] data = zooKeeper.getData("hdfs", true, stat);
System.out.println(new String(data));
System.out.println(stat.getCtime());
System.out.println(stat.getVersion());
}catch (Exception e){
e.printStackTrace();
}
}
}
二、Zookeeper伪分布式安装
1.预备工作
1.一台服务器
2.伪分布式zookeeper 1,2,3
3.创建zookeeper目录结构
zoo_1/2/3 用于存放节点的快照
zoo_2_dataLog 用于存放节点的事务日志
4.将zookeeper的压缩包解压到此处并改名为zookeeper-3.6.3
5.配置环境变量:vim /etc/profile
添加 export ZK_HOME=/zxy/apps/zookeeper/zookeeper-3.6.3
export PATH=$PATH:$ZK_HOME/bin
刷新资源 source /etc/profile
2.分配myid
#server1
echo "1" > /zxy/apps/zookeeper/zoo_1/myid
#server2
echo "2" > /zxy/apps/zookeeper/zoo_1/myid
#server3
echo "3" > /zxy/apps/zookeeper/zoo_1/myid
3.创建不同节点配置文件
[root@hadoop_zxy conf]# cp zoo_sample.cfg zoo1.cfg
[root@hadoop_zxy conf]# cp zoo_sample.cfg zoo2.cfg
[root@hadoop_zxy conf]# cp zoo_sample.cfg zoo3.cfg
[root@hadoop_zxy conf]# ls
configuration.xsl log4j.properties zoo1.cfg zoo2.cfg zoo3.cfg zoo_sample.cfg
4.修改配置文件内容
(三个节点的配置文件只有dataDir、dataLogDir、clientPort不同)
vim zoo1.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zxy/apps/zookeeper/zoo_1
dataLogDir=/zxy/apps/zookeeper/zoo_1_dataLog
clientPort=2181
server.1=hadoop_zxy:2887:3887
server.2=hadoop_zxy:2888:3888
server.3=hadoop_zxy:2889:3889
vim zoo2.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zxy/apps/zookeeper/zoo_2
dataLogDir=/zxy/apps/zookeeper/zoo_2_dataLog
clientPort=2182
server.1=hadoop_zxy:2887:3887
server.2=hadoop_zxy:2888:3888
server.3=hadoop_zxy:2889:3889
vim zoo3.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zxy/apps/zookeeper/zoo_3
dataLogDir=/zxy/apps/zookeeper/zoo_3_dataLog
clientPort=2183
server.1=hadoop_zxy:2887:3887
server.2=hadoop_zxy:2888:3888
server.3=hadoop_zxy:2889:3889
5.启动zookeeper
[root@hadoop_zxy zookeeper-3.6.3]#
bin/zkServer.sh start zoo1.cfg
bin/zkServer.sh start zoo2.cfg
bin/zkServer.sh start zoo3.cfg
6.查看进程(zookeeper已正常)
[root@hadoop_zxy zookeeper-3.6.3]# jps
29858 QuorumPeerMain
29715 QuorumPeerMain
19364 Jps
29977 QuorumPeerMain
[root@hadoop_zxy zookeeper-3.6.3]#
7.查看节点的leader和follower(半数原则)
[root@hadoop_zxy zookeeper-3.6.3]# bin/zkServer.sh status conf/zoo1.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo1.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
[root@hadoop_zxy zookeeper-3.6.3]# bin/zkServer.sh status conf/zoo2.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo2.cfg
Client port found: 2182. Client address: localhost. Client SSL: false.
Mode: leader
[root@hadoop_zxy zookeeper-3.6.3]# bin/zkServer.sh status conf/zoo3.cfg
ZooKeeper JMX enabled by default
Using config: conf/zoo3.cfg
Client port found: 2183. Client address: localhost. Client SSL: false.
Mode: follower
8.使用脚本管理(拓展)
vim start-zookeeper.sh
#!/bin/bash
# filename:start-zookeeper.sh
# autho:zxy
# date:2022-01-12
# KAFKA的安装路径
ZOOKEEPER_HOME=/zxy/apps/zookeeper/zookeeper-3.6.3
# 接受参数
CMD=$1
## 帮助函数
usage() {
echo "usage:"
echo "start-zookeeper.sh z1/z2/z3"
echo "description:"
echo " z1:start zoo_1.cfg"
echo " z2:start zoo_2.cfg"
echo " z3:stop zoo_3.cfg"
exit 0
}
if [ ${CMD} == "z1" ];then
# 启动zookeeper1服务
sh $ZOOKEEPER_HOME/bin/zkServer.sh start $ZOOKEEPER_HOME/conf/zoo1.cfg
elif [ ${CMD} == "z2" ];then
# 启动zookeeper2服务
sh $ZOOKEEPER_HOME/bin/zkServer.sh start $ZOOKEEPER_HOME/conf/zoo2.cfg
elif [ ${CMD} == "z3" ];then
# 启动zookeeper3服务
sh $ZOOKEEPER_HOME/bin/zkServer.sh start $ZOOKEEPER_HOME/conf/zoo3.cfg
else
usage
fi
三、Zookeeper连接超时问题,与拒绝连接问题解决
- 代码异常
- 处理方式
1.检查并关闭防火墙
2.检查vim /etc/hosts
端口冲突
删除第一行的127.0.0.1