ZooKeeper的客户端库Curator

    正常来说zk的内容到上一篇就可以结束了，但为了更好地使用zk，Curator是无法回避的话题。Curator是Netflix捐献给Apache的一套ZooKeeper客户端类库，提供了强大而简洁的封装，大大减少使用zk时重复发明轮子的情况。不客气的说，国内的Java程序员99%都是辣鸡，代码都是调对的而不是写对的，还特别喜欢重复发明轮子。相比之下，国外开源社区的代码质量就高得多，久经考验，即使有bug也基本可以做到明确可控的。只要你的应用适合这个层面的封装，就使用这个层面的开源代码，而不要使用更底层的，可以大大减少问题。

    当然我在使用Curator时还遇到一个问题（不能怪它的Committer），就是适配华X的hadoop版本时，由于华X的研发无耻地将ZooKeeper源码中的包名org.apache改为了com.huaxxx（其余代码几乎未改），使得社区编译好的Curator jar包不能直接使用，只能自己重新编译。适配CDH版本下的ZooKeeper则没有这种问题。

1 基本API

    创建一个Curator客户端非常简单，除了设置重试参数外，其余都可以用默认值。当然，也可以用工厂模式来设置更加复杂的参数。两种代码如下：

public class CreateClientExamples
{
    public static CuratorFramework createSimple(String connectionString)
    {
        // these are reasonable arguments for the ExponentialBackoffRetry. The first
        // retry will wait 1 second - the second will wait up to 2 seconds - the
        // third will wait up to 4 seconds.
        ExponentialBackoffRetry retryPolicy = new ExponentialBackoffRetry(1000, 3);

        // The simplest way to get a CuratorFramework instance. This will use default values.
        // The only required arguments are the connection string and the retry policy
        return CuratorFrameworkFactory.newClient(connectionString, retryPolicy);
    }

    public static CuratorFramework  createWithOptions(String connectionString, RetryPolicy retryPolicy, int connectionTimeoutMs, int sessionTimeoutMs)
    {
        // using the CuratorFrameworkFactory.builder() gives fine grained control
        // over creation options. See the CuratorFrameworkFactory.Builder javadoc
        // details
        return CuratorFrameworkFactory.builder()
            .connectString(connectionString)
            .retryPolicy(retryPolicy)
            .connectionTimeoutMs(connectionTimeoutMs)
            .sessionTimeoutMs(sessionTimeoutMs)
            // etc. etc.
            .build();
    }
}

    Curator使用了流畅式语法，熟悉scala或JAVA8函数式语法的同志会比较容易接受。基本的CRUD操作的示例代码如下：

public class CrudExamples
{
    public static void      create(CuratorFramework client, String path, byte[] payload) throws Exception
    {
        // this will create the given ZNode with the given data
        client.create().forPath(path, payload);
    }

    public static void      createEphemeral(CuratorFramework client, String path, byte[] payload) throws Exception
    {
        // this will create the given EPHEMERAL ZNode with the given data
        client.create().withMode(CreateMode.EPHEMERAL).forPath(path, payload);
    }

    public static String    createEphemeralSequential(CuratorFramework client, String path, byte[] payload) throws Exception
    {
        // this will create the given EPHEMERAL-SEQUENTIAL ZNode with the given data using Curator protection.

        /*
            Protection Mode:

            It turns out there is an edge case that exists when creating sequential-ephemeral nodes. The creation
            can succeed on the server, but the server can crash before the created node name is returned to the
            client. However, the ZK session is still valid so the ephemeral node is not deleted. Thus, there is no
            way for the client to determine what node was created for them.

            Even without sequential-ephemeral, however, the create can succeed on the sever but the client (for various
            reasons) will not know it. Putting the create builder into protection mode works around this. The name of
            the node that is created is prefixed with a GUID. If node creation fails the normal retry mechanism will
            occur. On the retry, the parent path is first searched for a node that has the GUID in it. If that node is
            found, it is assumed to be the lost node that was successfully created on the first try and is returned to
            the caller.
         */
        return client.create().withProtection().withMode(CreateMode.EPHEMERAL_SEQUENTIAL).forPath(path, payload);
    }

    public static void      setData(CuratorFramework client, String path, byte[] payload) throws Exception
    {
        // set data for the given node
        client.setData().forPath(path, payload);
    }

    public static void      setDataAsync(CuratorFramework client, String path, byte[] payload) throws Exception
    {
        // this is one method of getting event/async notifications
        CuratorListener listener = new CuratorListener()
        {
            @Override
            public void eventReceived(CuratorFramework client, CuratorEvent event) throws Exception
            {
                // examine event for details
            }
        };
        client.getCuratorListenable().addListener(listener);

        // set data for the given node asynchronously. The completion notification
        // is done via the CuratorListener.
        client.setData().inBackground().forPath(path, payload);
    }

    public static void      setDataAsyncWithCallback(CuratorFramework client, BackgroundCallback callback, String path, byte[] payload) throws Exception
    {
        // this is another method of getting notification of an async completion
        client.setData().inBackground(callback).forPath(path, payload);
    }

    public static void      delete(CuratorFramework client, String path) throws Exception
    {
        // delete the given node
        client.delete().forPath(path);
    }

    public static void      guaranteedDelete(CuratorFramework client, String path) throws Exception
    {
        // delete the given node and guarantee that it completes

        /*
            Guaranteed Delete

            Solves this edge case: deleting a node can fail due to connection issues. Further, if the node was
            ephemeral, the node will not get auto-deleted as the session is still valid. This can wreak havoc
            with lock implementations.


            When guaranteed is set, Curator will record failed node deletions and attempt to delete them in the
            background until successful. NOTE: you will still get an exception when the deletion fails. But, you
            can be assured that as long as the CuratorFramework instance is open attempts will be made to delete
            the node.
         */

        client.delete().guaranteed().forPath(path);
    }

    public static List<String> watchedGetChildren(CuratorFramework client, String path) throws Exception
    {
        /**
         * Get children and set a watcher on the node. The watcher notification will come through the
         * CuratorListener (see setDataAsync() above).
         */
        return client.getChildren().watched().forPath(path);
    }

    public static List<String> watchedGetChildren(CuratorFramework client, String path, Watcher watcher) throws Exception
    {
        /**
         * Get children and set the given watcher on the node.
         */
        return client.getChildren().usingWatcher(watcher).forPath(path);
    }
}

    各种API的详细参数就不介绍了。http://curator.apache.org/apidocs/index.html  这个页面下全有。

2 事件和监听器

    使用Curator库后，各种对zk的操作都会触发由Curator管理的事件。描述事件的类为CuratorEvent，其下定义了以下事件类型：CREATE, DELETE, EXISTS, GET_DATA, SET_DATA, CHILDREN, SYNC, GET_ACL, SET_ACL, TRANSACTION, WATCHED, GET_CONFIG, RECONFIG。

    基于事件的概念，有了Curator的监听器，它和ZooKeeper本身概念中的“监视”是有一定区别的。ZooKeeper的监视API采用链式调用和回调方法，任何watcher被设置后，均需要设置其回调方法，每个回调方法均需要有回调实现，也就是说这种处理器是“每事件一个”的。而使用Curator后，监视点通知、异常等均被封装为CuratorEvent，只需实现各类Listener接口的事件处理方法，并在client上注册监听器即可，也就是说这种处理器是“每客户端一个”的。其中一种接口是PathChildrenCacheListener，可用于实现对子目录各种操作的监听，示例代码如下：

/**
 * An example of the PathChildrenCache. The example "harness" is a command processor
 * that allows adding/updating/removed nodes in a path. A PathChildrenCache keeps a
 * cache of these changes and outputs when updates occurs.
 */
public class PathCacheExample
{
    private static final String     PATH = "/example/cache";

    public static void main(String[] args) throws Exception
    {
        TestingServer       server = new TestingServer();
        CuratorFramework    client = null;
        PathChildrenCache   cache = null;
        try
        {
            client = CuratorFrameworkFactory.newClient(server.getConnectString(), new ExponentialBackoffRetry(1000, 3));
            client.start();

            // in this example we will cache data. Notice that this is optional.
            cache = new PathChildrenCache(client, PATH, true);
            cache.start();

            processCommands(client, cache);
        }
        finally
        {
            CloseableUtils.closeQuietly(cache);
            CloseableUtils.closeQuietly(client);
            CloseableUtils.closeQuietly(server);
        }
    }

    private static void addListener(PathChildrenCache cache)
    {
        // a PathChildrenCacheListener is optional. Here, it's used just to log changes
        PathChildrenCacheListener listener = new PathChildrenCacheListener()
        {
            @Override
            public void childEvent(CuratorFramework client, PathChildrenCacheEvent event) throws Exception
            {
                switch ( event.getType() )
                {
                    case CHILD_ADDED:
                    {
                        System.out.println("Node added: " + ZKPaths.getNodeFromPath(event.getData().getPath()));
                        break;
                    }

                    case CHILD_UPDATED:
                    {
                        System.out.println("Node changed: " + ZKPaths.getNodeFromPath(event.getData().getPath()));
                        break;
                    }

                    case CHILD_REMOVED:
                    {
                        System.out.println("Node removed: " + ZKPaths.getNodeFromPath(event.getData().getPath()));
                        break;
                    }
                }
            }
        };
        cache.getListenable().addListener(listener);
    }

    private static void processCommands(CuratorFramework client, PathChildrenCache cache) throws Exception
    {
        // More scaffolding that does a simple command line processor

        printHelp();

        List<ExampleServer> servers = Lists.newArrayList();
        try
        {
            addListener(cache);

            BufferedReader  in = new BufferedReader(new InputStreamReader(System.in));
            boolean         done = false;
            while ( !done )
            {
                System.out.print("> ");

                String      line = in.readLine();
                if ( line == null )
                {
                    break;
                }

                String      command = line.trim();
                String[]    parts = command.split("\\s");
                if ( parts.length == 0 )
                {
                    continue;
                }
                String      operation = parts[0];
                String      args[] = Arrays.copyOfRange(parts, 1, parts.length);

                if ( operation.equalsIgnoreCase("help") || operation.equalsIgnoreCase("?") )
                {
                    printHelp();
                }
                else if ( operation.equalsIgnoreCase("q") || operation.equalsIgnoreCase("quit") )
                {
                    done = true;
                }
                else if ( operation.equals("set") )
                {
                    setValue(client, command, args);
                }
                else if ( operation.equals("remove") )
                {
                    remove(client, command, args);
                }
                else if ( operation.equals("list") )
                {
                    list(cache);
                }

                Thread.sleep(1000); // just to allow the console output to catch up
            }
        }
        finally
        {
            for ( ExampleServer server : servers )
            {
                CloseableUtils.closeQuietly(server);
            }
        }
    }

    private static void list(PathChildrenCache cache)
    {
        if ( cache.getCurrentData().size() == 0 )
        {
            System.out.println("* empty *");
        }
        else
        {
            for ( ChildData data : cache.getCurrentData() )
            {
                System.out.println(data.getPath() + " = " + new String(data.getData()));
            }
        }
    }

    private static void remove(CuratorFramework client, String command, String[] args) throws Exception
    {
        if ( args.length != 1 )
        {
            System.err.println("syntax error (expected remove <path>): " + command);
            return;
        }

        String      name = args[0];
        if ( name.contains("/") )
        {
            System.err.println("Invalid node name" + name);
            return;
        }
        String      path = ZKPaths.makePath(PATH, name);

        try
        {
            client.delete().forPath(path);
        }
        catch ( KeeperException.NoNodeException e )
        {
            // ignore
        }
    }

    private static void setValue(CuratorFramework client, String command, String[] args) throws Exception
    {
        if ( args.length != 2 )
        {
            System.err.println("syntax error (expected set <path> <value>): " + command);
            return;
        }

        String      name = args[0];
        if ( name.contains("/") )
        {
            System.err.println("Invalid node name" + name);
            return;
        }
        String      path = ZKPaths.makePath(PATH, name);

        byte[]      bytes = args[1].getBytes();
        try
        {
            client.setData().forPath(path, bytes);
        }
        catch ( KeeperException.NoNodeException e )
        {
            client.create().creatingParentContainersIfNeeded().forPath(path, bytes);
        }
    }

    private static void printHelp()
    {
        System.out.println("An example of using PathChildrenCache. This example is driven by entering commands at the prompt:\n");
        System.out.println("set <name> <value>: Adds or updates a node with the given name");
        System.out.println("remove <name>: Deletes the node with the given name");
        System.out.println("list: List the nodes/values in the cache");
        System.out.println("quit: Quit the example");
        System.out.println();
    }
}

3 菜谱

    所谓菜谱，是一系列分布式原语的实现，ZooKeeper本身只提供了基础API，而未提供原语的具体实现，就是为了将实现的主动权交给上层类库或应用。当然，ZooKeeper的文档中描述了ZooKeeper可以实现的各种分布式原语，Curator实现了其中的绝大部份（除了两阶段提交），列表如下：

选举：

• 主闩锁
• 主选举

锁：

• 共享重入锁
• 共享锁
• 共享重入读写锁
• 共享信号量
• 多共享锁

栅栏：

• 栅栏
• 双栅栏

计数器：

• 共享技术器
• 分布式Atomic Long类型

缓存：

• 路径缓存
• 节点缓存
• 树缓存

节点：

• 持久临时节点
• 群组成员

队列：

• 分布式队列
• 分布式ID队列
• 分布式优先级队列
• 分布式延迟队列
• 简单分布式队列

    光列名字我都有点晕，具体每种是什么意思，怎么用的，远超本文的范围，就不展开讲了。分布式原语实在是博大精深，zk也足够强大，这就是为什么简历上写精通什么的都有，就是没见过精通分布式事务或者zk的。

    前一节的例子已经展示了路径缓存，这里再展示一下主选举的代码，可以看到，代码本身乏善可陈，因为Curator为我们完全隐藏了分布式原语的细节，我们完全不用关心主选举算法、临时节点、监视这些zk里的概念，只需要关注应用本身。

/**
 * An example leader selector client. Note that {@link LeaderSelectorListenerAdapter} which
 * has the recommended handling for connection state issues
 */
public class ExampleClient extends LeaderSelectorListenerAdapter implements Closeable
{
    private final String name;
    private final LeaderSelector leaderSelector;
    private final AtomicInteger leaderCount = new AtomicInteger();

    public ExampleClient(CuratorFramework client, String path, String name)
    {
        this.name = name;

        // create a leader selector using the given path for management
        // all participants in a given leader selection must use the same path
        // ExampleClient here is also a LeaderSelectorListener but this isn't required
        leaderSelector = new LeaderSelector(client, path, this);

        // for most cases you will want your instance to requeue when it relinquishes leadership
        leaderSelector.autoRequeue();
    }

    public void start() throws IOException
    {
        // the selection for this instance doesn't start until the leader selector is started
        // leader selection is done in the background so this call to leaderSelector.start() returns immediately
        leaderSelector.start();
    }

    @Override
    public void close() throws IOException
    {
        leaderSelector.close();
    }

    @Override
    public void takeLeadership(CuratorFramework client) throws Exception
    {
        // we are now the leader. This method should not return until we want to relinquish leadership

        final int         waitSeconds = (int)(5 * Math.random()) + 1;

        System.out.println(name + " is now the leader. Waiting " + waitSeconds + " seconds...");
        System.out.println(name + " has been leader " + leaderCount.getAndIncrement() + " time(s) before.");
        try
        {
            Thread.sleep(TimeUnit.SECONDS.toMillis(waitSeconds));
        }
        catch ( InterruptedException e )
        {
            System.err.println(name + " was interrupted.");
            Thread.currentThread().interrupt();
        }
        finally
        {
            System.out.println(name + " relinquishing leadership.\n");
        }
    }
}

public class LeaderSelectorExample
{
    private static final int        CLIENT_QTY = 10;

    private static final String     PATH = "/examples/leader";

    public static void main(String[] args) throws Exception
    {
        // all of the useful sample code is in ExampleClient.java

        System.out.println("Create " + CLIENT_QTY + " clients, have each negotiate for leadership and then wait a random number of seconds before letting another leader election occur.");
        System.out.println("Notice that leader election is fair: all clients will become leader and will do so the same number of times.");

        List<CuratorFramework>  clients = Lists.newArrayList();
        List<ExampleClient>     examples = Lists.newArrayList();
        TestingServer           server = new TestingServer();
        try
        {
            for ( int i = 0; i < CLIENT_QTY; ++i )
            {
                CuratorFramework    client = CuratorFrameworkFactory.newClient(server.getConnectString(), new ExponentialBackoffRetry(1000, 3));
                clients.add(client);

                ExampleClient       example = new ExampleClient(client, PATH, "Client #" + i);
                examples.add(example);

                client.start();
                example.start();
            }

            System.out.println("Press enter/return to quit\n");
            new BufferedReader(new InputStreamReader(System.in)).readLine();
        }
        finally
        {
            System.out.println("Shutting down...");

            for ( ExampleClient exampleClient : examples )
            {
                CloseableUtils.closeQuietly(exampleClient);
            }
            for ( CuratorFramework client : clients )
            {
                CloseableUtils.closeQuietly(client);
            }

            CloseableUtils.closeQuietly(server);
        }
    }
}