Zookeeper 3.5.7学习记录(二)——命令行、JavaAPI和分布式锁

Zookeeper 3.5.7学习记录(二)——命令行、JavaAPI和分布式锁

对应课程

【尚硅谷】大数据技术之Zookeeper 3.5.7版本教程

集群的统一启动和停止脚本

由于分别在各个主机上启动和停止Zookeeper的过程十分繁琐，因此，编写一个Shell脚本统一的控制开关十分必要。首先，创建一个"zk.sh"的脚本，我选择主机"hadoop101"，并把目录定在"/opt/module/zookeeper-3.5.7/bin"。

vim zh.sh

由于用到了sshpass登录，因而，先下载sshpass的包：

yum install sshpass

最好选择在Notepad++中编写，再传到虚拟机上，编写脚本"zk.sh"。

#!/bin/bash

case $1 in
"start"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo -------------- zookeeper $i 启动 ---------------
                sshpass -p 123456 ssh -o StrictHostKeyChecking=no $i "/opt/module/zookeeper-3.5.7/bin/sh zkServer.sh start"
        done
}
;;
"stop"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo -------------- zookeeper $i 停止 ---------------
                sshpass -p 123456 ssh -o StrictHostKeyChecking=no $i "/opt/module/zookeeper-3.5.7/bin/sh zkServer.sh stop"
        done
}
;;
"status"){
        for i in hadoop101 hadoop102 hadoop103
        do
                echo -------------- zookeeper $i 状态 ---------------
                sshpass -p 123456 ssh -o StrictHostKeyChecking=no $i "/opt/module/zookeeper-3.5.7/bin/sh zkServer.sh status"
        done
}
;;
esac

注意：在Notepad++中，右下角选择"Unix(LF)"。否则，会报错：'/r’无法识别。

如果想要更方便地启动zookeeper集群，可以配置环境变量：

vim /etc/profile

在底部添加：

export PATH=$PATH:/opt/module/zookeeper-3.5.7/bin

然后:wq保存退出。读取并执行环境变量：

source /etc/profile

统一启动zookeeper集群(hadoop101，hadoop102，hadoop103)：

sh zk.sh start

[root@hadoop101 etc]# sh zk.sh start
-------------- zookeeper hadoop101 启动 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED
-------------- zookeeper hadoop102 启动 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED
-------------- zookeeper hadoop103 启动 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Starting zookeeper … STARTED

查看状态：

[root@hadoop101 etc]# sh zk.sh status
-------------- zookeeper hadoop101 状态 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
-------------- zookeeper hadoop102 状态 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
-------------- zookeeper hadoop103 状态 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

停止集群：

sh zk.sh stop

[root@hadoop101 etc]# sh zk.sh stop
-------------- zookeeper hadoop101 停止 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Stopping zookeeper … STOPPED
-------------- zookeeper hadoop102 停止 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Stopping zookeeper … STOPPED
-------------- zookeeper hadoop103 停止 ---------------
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/…/conf/zoo.cfg
Stopping zookeeper … STOPPED

客户端命令行操作

使用ls命令来查看路径"path"下的子节点，附带"-w"表示监听子节点的变化，"-s"可查看次级信息。

ls -s -w path

创建子节点及其存储的信息，附带"-s"表示含有序列，"-e"表示临时节点（重启或超时会消失）。

create -s -e path info

查看节点信息，附带"-w"表示监听节点内容的变化，"-s"可查看次级信息。

get -w -s path

设置节点具体的值。

set path info

查看节点状态。

stat path

删除节点。

delete path

删除该节点下的所有子节点。

deleteall path

JavaAPI操作

使用SpringBoot框架，然后额外添加依赖：

<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.6.0</version>
</dependency>

添加配置文件ZKConfig.java，将ZK客户端放入IOC容器中。

package com.jd.zookeeper.conf;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.io.IOException;
import java.util.List;

@Configuration
public class ZKConfig {

    private ZooKeeper zooKeeper = null;

    @Bean
    public ZooKeeper getZooKeeperCli() {
        String connectString = "hadoop101:2181,hadoop102:2181,hadoop103:2181";
        int sessionTimeout = 10000;
        try {
            zooKeeper = new ZooKeeper(connectString, sessionTimeout, new Watcher() {
                @Override
                public void process(WatchedEvent watchedEvent) {
                    List<String> children = null;
                    try {
                        System.out.println("--------------------------------------");
                        children = zooKeeper.getChildren("/", true);
                        children.forEach((child)->{
                            System.out.println(child);
                        });
                        System.out.println("--------------------------------------");
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }
            });
        } catch (IOException e) {
            e.printStackTrace();
        }
        return zooKeeper;
    }
}

在测试类ZookeeperApplicationTests中，测试JavaAPI：

package com.jd.zookeeper;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.test.context.SpringBootTest;

import java.util.List;

@SpringBootTest
class ZookeeperApplicationTests {

	@Autowired
	private ZooKeeper zooKeeper;

	//创造节点
	@Test
	void createNode() throws InterruptedException, KeeperException {
		String nodeCreated = zooKeeper.create("/sanguo", "hanxiandi".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
		System.out.println(nodeCreated);
	}

	//查询子节点
	@Test
	void getChildrenNode() throws InterruptedException, KeeperException {
		List<String> children = zooKeeper.getChildren("/", true);
		Thread.sleep(Long.MAX_VALUE);
	}

	//判断该节点是否存在
	@Test
	void exist() throws InterruptedException, KeeperException {
		Stat stat = zooKeeper.exists("/sanguo", false);
		System.out.println(stat == null ? "not exist" : "exist");
	}

}

Zookeeper分布式锁

当一个线程尝试访问一个资源时，会先尝试获取它的锁，在获取到锁之后，保持对该资源的独占，进行读写操作。直至该线程释放锁，其余线程才能获取锁，进而操作资源。

通过这个锁机制，保证了分布式系统中，能够有序地访问该资源，我们把这个适用于分布式情况下的锁称作“分布式锁”。

使用Curator框架实现分布式锁：

导入jar包，

<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-framework</artifactId>
    <version>4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-recipes</artifactId>
    <version>4.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-client</artifactId>
    <version>4.3.0</version>
</dependency>

编写实例CuratorLockTest：

package com.jd.zookeeper.case3;


import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.apache.curator.retry.ExponentialBackoffRetry;

public class CuratorLockTest {

    public static void main(String[] args) {
        //创建分布式锁1
        InterProcessMutex lock1 = new InterProcessMutex(getCuratorFramework(), "/locks");

        //创建分布式锁2
        InterProcessMutex lock2 = new InterProcessMutex(getCuratorFramework(), "/locks");

        //创建多线程
        new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    lock1.acquire();
                    System.out.println("线程1  获取到锁");

                    lock1.acquire();
                    System.out.println("线程1  再次获取到锁");

                    Thread.sleep(5000);

                    lock1.release();
                    System.out.println("线程1  释放锁");

                    lock1.release();
                    System.out.println("线程1  再次释放锁");
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }).start();

        new Thread(new Runnable() {
            @Override
            public void run() {
                try {
                    lock2.acquire();
                    System.out.println("线程2  获取到锁");

                    lock2.acquire();
                    System.out.println("线程2  再次获取到锁");

                    Thread.sleep(5000);

                    lock2.release();
                    System.out.println("线程2  释放锁");

                    lock2.release();
                    System.out.println("线程2  再次释放锁");
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }).start();
    }

    // policy：断线重连规则，最多尝试3次，间隔3秒
    // connectionTimeout：连接超时时长，客户端尝试与服务器连接的最大时长
    // sessionTimeout：会话超时时长，超过该时长服务器没有收到心跳，则删除会话
    private static CuratorFramework getCuratorFramework() {
        RetryPolicy policy = new ExponentialBackoffRetry(3000,3);
        CuratorFramework client = CuratorFrameworkFactory.builder().connectString("hadoop101:2181,hadoop102:2181,hadoop103:2181")
                .connectionTimeoutMs(5000)
                .sessionTimeoutMs(5000)
                .retryPolicy(policy).build();
        //启动客户端
        client.start();
        System.out.println("zk启动成功");
        return client;
    }
}

Zookeeper控制台大量输出DEBUG级日志问题解决

导入依赖：

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-logging</artifactId>
</dependency>

在src/main/resources目录下编写配置文件logback.xml：

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <!--格式化输出：%d表示日期，%thread表示线程名，%-5level：级别靠左显示5个字符宽度，%logger：日志的类名，%msg：日志消息，%n是换行符 -->
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} --- %msg%n</pattern>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

在application.yml配置文件中指定路径：

logging:
  config: classpath:logback.xml