6 Zookeeper开荒

最新推荐文章于 2021-10-24 19:02:36 发布

oifengo

最新推荐文章于 2021-10-24 19:02:36 发布

阅读量320

点赞数 1

分类专栏： # 爬梯

本文链接：https://blog.csdn.net/weixin_39381833/article/details/108070750

版权

爬梯专栏收录该内容

47 篇文章 0 订阅

订阅专栏

1 Zookeeper架构

Zookeeper ： open source 、 distributed 、为分布式应用提供协调服务的Apache项目。

存储 & 管理数据
接收观察者的注册
数据变化讲负责通知已经在Zookeeper上注册的那些观察者做出反应。

在这里插入图片描述

服务端启动时去注册信息(创建时都是临时节点)
获取到当前在线服务列表，并且注册监听
假如有服务节点下线
服务节点下线时间通告
process() 重新再去获取服务器列表，并注册监听

1.1 ZK特点

在这里插入图片描述

一个leader ，多个Follwer 组成的集群
集群只要有半数以上的节点存货，ZK集群就能正常服务
全局数据一致性：每个Server保存一份相同的数据副本，Client无论链接哪个数据都是一致的
更新请求顺序进行，同一个Client的更新请求按照发出顺序依次执行
数据更新原子性，一次数据更新要么成功要么失败
实时性，在一定时间范围内，Client能读到最新数据

1.2 ZK数据结构

ZK数据模型的结构与Unix文件系统很类似，整体上可以看作是一棵树，每个节点叫做一个ZNode。
每一个ZNode默认能够存储 1MB 的数据，每个 ZNode都可以通过其路径唯一标识。

在这里插入图片描述

1.3 应用场景

统一命名服务
统一配置管理
统一集群管理
服务器节点动态上下

1.3.1 统一命名服务

分布式环境下，经常需要对应用 & 服务进行统一的命名，便于识别
例如：IP不容易记住但是域名更容易记住。
在这里插入图片描述

1.3.2 统一配置管理

分布式环境下配置文件的同步很常见

一个集群当中所有节点的配置信息都是一致的，例如Kafka
对配置文件修改后，希望能够快速同步到各个节点上

配置文件交互可由ZK实现

将配置信息写入ZK上的一个Znode
每个客户端服务器监听这个Znode
一旦Znode中的数据被修改，ZK将通知各个客户端

1.3.3 统一集群管理

分布式环境中，实时监控节点状态
- 根据节点状态做出调整
ZK可以实现实时监控节点状态变化
- 将节点信息写入到ZK上的一个ZNode
- 监听这个ZNode获取它的实时状态变化

1.3.4 服务动态上下线

在这里插入图片描述

软负载均衡

记录每台服务器的访问数，让访问数最小的服务器去处理最新的客户端请求

在这里插入图片描述

1.4 选举机制

半数机制：集群中半数以上的机器存活，集群仍然可用。所以ZK适合安装奇数
ZK虽然在配置文件中并没有设定 Master 和 Slave 。但是工作中只有一个节点是Leader 其他的为Follower
选举过程
- Server 1 启动，只有一个发出的保温没有任何相应，选举状态一致处于LOOKING
- Server2 启动，和 1 进行通信选举，大概率 2 胜出(都没有历史数据) 但是没有超半数票，所以 1 2 都处于LOOKING
- Server 3 启动，过半数票 3 成为Leader
- Server 4 启动，已经有了Leader 直接成为Follower
- Server 5 启动， follower.

2 Zookeeper 安装

tar -zxvf zookeeper-3.4.10.tar.gz -C /opt/module/

cp ./conf/zoo_sample.cfg zoo.cfg

由于/tmp下Linux会定时清理所以要更改dataDir

dataDir=/opt/module/zookeeper-3.4.10/zkData

在ZK安装目录下创建zkData

mkdir zkData

配置进环境变量

#env
export HADOOP_HOME=/home/ifeng/app/hadoop
export HIVE_HOME=/home/ifeng/app/hive
export ZOOKEEPER_HOME=/home/ifeng/app/zookeeper

export PATH=${ZOOKEEPER_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HIVE_HOME}/bin:$PATH

2.2 启动Zookeeper

 bin/zkServer.sh start

在这里插入图片描述

2.3 查看Zookeeper状态

bin/zkServer.sh status

在这里插入图片描述

启动后查看dataDir ，会有pid文件存放端口 (kill的时候直接从这里读取)
在这里插入图片描述

2.4 启动客户端

 bin/zkCli.sh

在这里插入图片描述

-- 退出
quit

2.5 停止Zookeeper

bin/zkServer.sh stop

2.6 zoo.cfg参数配置解读

tickTime =2000**

通信心跳数，ZK服务器 & Client之间的通信心跳时间，单位毫秒

也就是每个tickTime时间就会发送一个心跳时间单位为毫秒

用于心跳机制，并且设置了最小的session超时时间为两倍的心跳时间

initLimit =10**

LF初始通信时限

Follower & Leader 之间的 初始连续时 能忍受的最多心跳数
用它来设定ZK服务器链接到 Leader的时限。

syncLimit =5**

LF 同步通信时限

Leader & Follower 之间最大相应时间

假如超过syncLimit * tickTime ，

Leader会认为 Follower死掉，服务列表删除此 Foller

dataDir

保存Zookeeper 中的数据

clientPort =2181

监听客户端链接的端口

2.7 客户端操作

目录结构(树形结构)
节点有唯一的标识

存储的数据有版本号
修改后 版本号 + 1

修改/删除可以指定版本号，版本号不匹配会报错

znode上的数据不宜过大，几k即可
znode是可以设置权限
znode可以设置监听器，当节点数据发生变化，可以通过监听器获取

Znode节点分两种类型：

临时：当前session有效，临时的不能再有子节点
永久：其他sessions也可以

四种形式

永久/持久
永久/持久带顺序编号
临时
…


[zk: localhost:2181(CONNECTED) 1] help
ZooKeeper -server host:port cmd args
        stat path [watch]
        set path data [version]
        ls path [watch]
        delquota [-n|-b] path
        ls2 path [watch]
        setAcl path acl
        setquota -n|-b val path
        history
        redo cmdno
        printwatches on|off
        delete path [version]
        sync path
        listquota path
        rmr path
        get path [watch]
        create [-s] [-e] path data acl
        addauth scheme auth
        quit
        getAcl path
        close
        connect host:port

创建节点create
通过 create 命令在根目录创建了node1节点，与它关联的字符串是"node1"

[zk: localhost:2181(CONNECTED) 7] create /node1 "node1"
Created /node1

通过 create 命令在根目录创建了node1节点，与它关联的内容是数字 123

[zk: localhost:2181(CONNECTED) 8] create /node1/node1.1 123
Created /node1/node1.1

更新节点数据set

[zk: localhost:2181(CONNECTED) 12] set /node1 "setnode1"
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x6
mtime = Wed Aug 19 05:30:10 EDT 2020
pZxid = 0x5
cversion = 1
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 1

获取节点信息get

[zk: localhost:2181(CONNECTED) 13] get /node1
"setnode1"
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x6
mtime = Wed Aug 19 05:30:10 EDT 2020
pZxid = 0x5
cversion = 1
dataVersion = 1  这里的版本由 0 ---- > 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 1

查看节点ls

[zk: localhost:2181(CONNECTED) 14] ls /
[zookeeper, node1]

ZK中的ls 与 linux 中的ls 类似，返回绝对路径path下的所有子节点信息(只列出1级不递归)

查看节点状态（stat）

[zk: localhost:2181(CONNECTED) 15] stat /node1
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x6
mtime = Wed Aug 19 05:30:10 EDT 2020
pZxid = 0x5
cversion = 1
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 1

查看节点状态和信息ls2

[zk: localhost:2181(CONNECTED) 16] ls2 /node1
[node1.1]
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x6
mtime = Wed Aug 19 05:30:10 EDT 2020
pZxid = 0x5
cversion = 1
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 1

ls2 = ls + stat
返回子节点列表 + 当前节点的stat信息

删除节点(delete)


[zk: localhost:2181(CONNECTED) 18] delete /node1/node1.1
[zk: localhost:2181(CONNECTED) 19]

多节点

# the port at which the clients will connect
clientPort=2181

server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890

3 Watch 监听

watch 不依赖于已有的目录，没有这个目录也可以监听

watch是一次性的第二次改变不会监听

get watch

[zk: localhost:2181(CONNECTED) 19] get /node1 watch
"setnode1"
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x6
mtime = Wed Aug 19 05:30:10 EDT 2020
pZxid = 0x8
cversion = 2
dataVersion = 1
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 10
numChildren = 0
[zk: localhost:2181(CONNECTED) 20] set /node1 "watchnode1"

WATCHER::

WatchedEvent state:SyncConnected type:NodeDataChanged path:/node1
cZxid = 0x4
ctime = Wed Aug 19 05:23:06 EDT 2020
mZxid = 0x9
mtime = Wed Aug 19 06:17:10 EDT 2020
pZxid = 0x8
cversion = 2
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 12
numChildren = 0
[zk: localhost:2181(CONNECTED) 21]

ls watch

[zk: localhost:2181(CONNECTED) 23] ls /node1 watch
[node1.2]
[zk: localhost:2181(CONNECTED) 24] create /node1/node1.3 123

WATCHER::
Created /node1/node1.3
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/node1

[zk: localhost:2181(CONNECTED) 25]

ls2 path [watch]

[zk: localhost:2181(CONNECTED) 26] ls2 /node1/node1.3 watch
[]
cZxid = 0xb
ctime = Wed Aug 19 06:18:31 EDT 2020
mZxid = 0xb
mtime = Wed Aug 19 06:18:31 EDT 2020
pZxid = 0xb
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0



[zk: localhost:2181(CONNECTED) 29] create /node1/node1.3/node1.3.1 5423

WATCHER::

WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/node1/node1.3
Created /node1/node1.3/node1.3.1
[zk: localhost:2181(CONNECTED) 30]

4.2 四字命令

[ifeng@ifeng root]$ echo ruok | nc localhost 2181
imok[ifeng@ifeng root]$ echo stat | nc localhost 2181
Zookeeper version: 3.4.5-cdh5.16.2--1, built on 06/03/2019 10:40 GMT
Clients:
 /0:0:0:0:0:0:0:1:55476[0](queued=0,recved=1,sent=0)
 /0:0:0:0:0:0:0:1:55044[1](queued=0,recved=1054,sent=1057)

Latency min/avg/max: 0/0/13
Received: 1064
Sent: 1066
Connections: 2
Outstanding: 0
Zxid: 0xe
Mode: standalone
Node count: 8
[ifeng@ifeng root]$

stat ruok

5 API调用

 <dependency>
        <groupId>org.apache.zookeeper</groupId>
        <artifactId>zookeeper</artifactId>
        <version>3.4.5-cdh5.16.2</version>
    </dependency>

package zookeeper;

import org.apache.zookeeper.*;
import org.junit.Before;
import org.junit.Test;

public class zkutils {

    private static String connectString = "ifeng:2181";
    private static int sessionTimeout = 2000;
    private ZooKeeper zkClient = null;

    @Before
    public void init() throws Exception {

        zkClient = new ZooKeeper(connectString, sessionTimeout, new Watcher() {

            @Override
            public void process(WatchedEvent event) {

                // 收到事件通知后的回调函数（用户的业务逻辑）
                System.out.println(event.getType() + "--" + event.getPath());

                // 再次启动监听
                try {
                    zkClient.getChildren("/", true);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });


    @After
    public void tearDown(){
        if(null != client) {
            client.close();
        }
    }



    }
    // 创建子节点
    @Test
    public void create() throws Exception {

        // 参数1：要创建的节点的路径； 参数2：节点数据 ； 参数3：节点权限 ；参数4：节点的类型
        String nodeCreated = zkClient.create("/ifeng", "ifengtest".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
    }


}

    @Test
    public void getChildren() throws Exception {

        List<String> children = zkClient.getChildren("/", true);

        for (String child : children) {
            System.out.println(child);
        }

        // 延时阻塞
        Thread.sleep(Long.MAX_VALUE);
    }

    // 判断znode是否存在
    @Test
    public void exist() throws Exception {

        Stat stat = zkClient.exists("/eclipse", false);
        Stat stat2 = zkClient.exists("/ifeng", false);

        System.out.println(stat == null ? "not exist" : "exist");
        System.out.println(stat2 == null ? "not exist" : "exist");
    }

6 Curator

高级API 调用Zookeeper

        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-framework</artifactId>
            <version>4.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.curator</groupId>
            <artifactId>curator-recipes</artifactId>
            <version>4.0.0</version>
        </dependency>

package curator;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.junit.Before;

public class curatorTest {

    CuratorFramework client = null;
    String zkQuorum = "ifeng:2181";
    String nodePath = "/ifeng";
    
    @Before
    public void setup(){
        RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000,5);
        
        client = CuratorFrameworkFactory.builder()
                .connectString(zkQuorum)
                .sessionTimeoutMs(10000)
                .retryPolicy(retryPolicy)
                .namespace("ifengdata-workspace")
                .build();
        client.start();
    }

}

    @Test
    public void testSetData() throws Exception {
        client.setData().forPath(nodePath, "ifengdata".getBytes());
    }

    @Test
    public void testGetData() throws Exception {
        Stat stat = new Stat();
        String data = new String(client.getData().storingStatIn(stat).forPath(nodePath));
        System.out.println(data);
        System.out.println(stat.getVersion());
    }

    @Test
    public void testExist()throws Exception {
        Stat stat = client.checkExists().forPath("/aad");
        System.out.println(stat);
    }


    @Test
    public void testGetChildren() throws Exception {
        List<String> children = client.getChildren().forPath(nodePath);
        for(String child : children) {
            System.out.println(child);
        }
    }

    @Test
    public void testDelete() throws Exception {
        client.delete()
                .deletingChildrenIfNeeded()
                .withVersion(10)
                .forPath(nodePath+"/c");
    }

7 企业案例

7.1 Curator实现永久监听

package curator;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.curator.framework.recipes.cache.NodeCacheListener;
import org.apache.curator.retry.ExponentialBackoffRetry;

public class CuratorNodeCache {

    public static void main(String[] args) throws Exception {

        CuratorFramework clinet = getClinet();
        String path = "/ifeng";
        NodeCache nodeCache = new NodeCache(clinet, path);
        nodeCache.start();
        nodeCache.getListenable().addListener(new NodeCacheListener() {
            @Override
            public void nodeChanged() throws Exception {
                System.out.println("触发监听");
                System.out.println("更新数据为:：" + new String(nodeCache.getCurrentData().getData()));
            }
        });

        clinet.setData().forPath(path,"123".getBytes());
        clinet.setData().forPath(path,"456".getBytes());
        clinet.setData().forPath(path,"789".getBytes());
        clinet.setData().forPath(path,"101".getBytes());
        clinet.setData().forPath(path,"112".getBytes());
        clinet.setData().forPath(path,"113".getBytes());
        Thread.sleep(15000);

    }

    private static CuratorFramework getClinet() {
        ExponentialBackoffRetry retryPolicy = new ExponentialBackoffRetry(1000, 3);
        CuratorFramework client = CuratorFrameworkFactory.builder()
                .connectString("ifeng:2181")
                .retryPolicy(retryPolicy)
                .sessionTimeoutMs(1000)
                .connectionTimeoutMs(3000)
                //.namespace("demo")
                .build();
        client.start();
        return client;
    }

}

在这里插入图片描述

7.2 原生API 递归删除创建

ZK只允许删除叶子节点，想要删除非叶子节点只能递归删除

package zookeeper;

import java.io.IOException;
import java.util.List;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;

/**

 *
 */
public class recursion {

    private static String connectString = "ifeng:2181";
    private static int sessionTimeout = 2000;
    private ZooKeeper zkClient = null;

    /**
     * main函数
     * @param args
     * @throws Exception
     */
    public static void main(String[] args) throws Exception {

        //调用rmr,删除所有目录
        rmr("/ifeng");
    }

    /**
     * 递归删除 因为zookeeper只允许删除叶子节点，如果要删除非叶子节点，只能使用递归
     * @param path
     * @throws IOException
     */
    public static void rmr(String path) throws Exception {
        ZooKeeper zk = getZookeeper();
        //获取路径下的节点
        List<String> children = zk.getChildren(path, false);
        for (String pathCd : children) {
            //获取父节点下面的子节点路径
            String newPath = "";
            //递归调用,判断是否是根节点
            if (path.equals("/")) {
                newPath = "/" + pathCd;
            } else {
                newPath = path + "/" + pathCd;
            }
            rmr(newPath);
        }
        //删除节点,并过滤zookeeper节点和 /节点
        if (path != null && !path.trim().startsWith("/zookeeper") && !path.trim().equals("/")) {
            zk.delete(path, -1);
            //打印删除的节点路径
            System.out.println("被删除的节点为：" + path);
        }
    }

    /**
     * 获取Zookeeper实例
     * @return
     * @throws IOException
     */
    public static ZooKeeper getZookeeper() throws IOException {
        ZooKeeper zookeeper = new ZooKeeper(connectString, sessionTimeout, new MyWatch());
        return zookeeper;
    }

}

在这里插入图片描述

递归创建

package zookeeper;

import org.apache.zookeeper.*;

import java.io.IOException;

public class recursionCreat {

    private static String connectString = "ifeng:2181";
    private static int sessionTimeout = 2000;
    private ZooKeeper zkClient = null;

    public void main(String[] args) throws Exception {
        Createss("/ifeng/ifeng01/ifeng02");
    }


    public void Createss(String path) throws Exception {

        ZooKeeper zk = getZookeeper();
        if(zk.exists(path, (Watcher) this) == null && path.length() > 0){
            String temp = path.substring(0,path.lastIndexOf("/"));
            Createss(temp);
            zk.create(path,null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
        }else{
            return;
        }


    }

    public static ZooKeeper getZookeeper() throws Exception {
        ZooKeeper zookeeper = new ZooKeeper(connectString, sessionTimeout, new MyWatch());
        return zookeeper;
    }
}

分布式锁

package ZKLock;

import javafx.scene.shape.Path;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs;
import org.junit.Before;
import org.junit.Test;
import org.junit.platform.commons.util.StringUtils;

import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;

public class zkTest implements Lock {

    CuratorFramework client = null;
    String zkQuorum = "ifeng:2181";
    String nodePath = "/lock";
    private static final String Z_NODE = "/LOCK";

    private volatile String beforePath;
    private volatile String path;

    @Before
    public void setup(){
        RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000,5);

        client = CuratorFrameworkFactory.builder()
                .connectString(zkQuorum)
                .sessionTimeoutMs(10000)
                .retryPolicy(retryPolicy)
                .namespace("ifengdata-workspace")
                .build();
        client.start();
    }

    @Test
    public void testCreateNode() throws Exception {
        client.create().creatingParentsIfNeeded()
                .withMode(CreateMode.PERSISTENT)
                .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE)
                .forPath(nodePath, "curator-ifeng".getBytes());
    }


    @Override
    public void lock() {

    }

    @Override
    public void lockInterruptibly() throws InterruptedException {

    }

    public synchronized boolean tryLock() {

        //第一次进来创建自己的临时节点
        if(StringUtils.isBlank(Z_NODE)){
            try {
                path = client.create().creatingParentContainersIfNeeded()
                        .withMode(CreateMode.PERSISTENT)
                        .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE)
                        .forPath(Z_NODE,"lock".getBytes());
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        //对节点排序
        //List<String> children = client.getChildren(Z_NODE);
        List<String> children = null;
        try {
            children = client.getChildren().forPath(Z_NODE);
        } catch (Exception e) {
            e.printStackTrace();
        }
        Collections.sort(children);

        //当前节点是最小节点就加锁成功
        if(path.equals(Z_NODE + "/" + children.get(0))) {
            System.out.println(" first ");
            return true;
        }else {
            //不是最小节点 ， 就找到前面的那一个 释放也同理
            int i = Collections.binarySearch(children, path.substring(Z_NODE.length() + 1));
            beforePath = Z_NODE + "/" + children.get(i - 1);
        }
        return false;
    }

    @Override
    public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {
        return false;
    }

    public void unlock(){
        client.delete(Z_NODE);
    }

    @Override
    public Condition newCondition() {
        return null;
    }

    public void waitForLock() throws Exception {

        byte[] content = client.getData()
                .usingWatcher(new Watcher() {
                    @Override
                    public void process(WatchedEvent watchedEvent) {
                        System.out.println("监听到nulock");
                    }
                }).forPath(Z_NODE);

        if(content != null){

        }

    }

    

}

oifengo

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
6 Zookeeper开荒

文章目录1 Zookeeper架构1.1 ZK特点1.2 ZK数据结构1.3 应用场景1.3.1 统一命名服务1.3.2 统一配置管理1.3.3 统一集群管理1.3.4 服务动态上下线软负载均衡1.4 选举机制1 Zookeeper架构Zookeeper ： open source 、 distributed 、为分布式应用提供协调服务的Apache项目。存储 & 管理数据接收观察者的注册数据变化讲负责通知已经在Zookeeper上注册的那些观察者做出反应。服务端启动时去注册
复制链接

扫一扫