ZooKeeper 8：请求处理逻辑与源码分析

prev:ZooKeeper 7：数据读写——原子广播协议ZAB

责任链模式

当客户端需要和 ZooKeeper 服务端进行相互协调通信时，首先要建立该客户端与服务端的连接会话，在会话成功创建后，ZooKeeper 服务端就可以接收来自客户端的请求操作了。
ZooKeeper集群在收到事务性会话请求后，主要依次进行四个部分的处理逻辑：预处理阶段、事务处理阶段、事务执行阶段、响应客户端。这种处理就像流水线一样，也是责任链模式的一种实现。
leader、follower、observer的责任链显然也是不同的。

预处理阶段

在预处理阶段，主要是分辨要处理的请求是否是事务性请求，比如创建节点、更新数据、删除节点、创建会话等，这些请求操作都是事务性请求，在执行成功后会对服务器上的数据造成影响。
预处理的相关逻辑是在PrepRequestProcessor类下。PrepRequestProcessor类里面Override了run()函数，起了一个线程，来从submittedRequests里面不停的获取request，submittedRequests是一个LinkedBlockingQueue<Request>类型的阻塞队列：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* PrepRequestProcessor.java
	* 137行
	*/
	@Override
    public void run() {
        LOG.info(String.format("PrepRequestProcessor (sid:%d) started, reconfigEnabled=%s", zks.getServerId(), zks.reconfigEnabled));
        try {
            while (true) {
                ServerMetrics.getMetrics().PREP_PROCESSOR_QUEUE_SIZE.add(submittedRequests.size());
                Request request = submittedRequests.take();
                ServerMetrics.getMetrics().PREP_PROCESSOR_QUEUE_TIME
                    .add(Time.currentElapsedTime() - request.prepQueueStartTime);
                if (LOG.isTraceEnabled()) {
                    long traceMask = ZooTrace.CLIENT_REQUEST_TRACE_MASK;
                    if (request.type == OpCode.ping) {
                        traceMask = ZooTrace.CLIENT_PING_TRACE_MASK;
                    }
                    ZooTrace.logRequest(LOG, traceMask, 'P', request, "");
                }
                if (Request.requestOfDeath == request) {
                    break;
                }

                request.prepStartTime = Time.currentElapsedTime();
                pRequest(request);
            }
        } catch (Exception e) {
            handleException(this.getName(), e);
        }
        LOG.info("PrepRequestProcessor exited loop!");
    }

这里调用了pRequest(request)函数来处理请求。pRequest()判断客户端发送的会话请求类型，并执行相应逻辑：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* PrepRequestProcessor.java
	* 772行
	*/
	protected void pRequest(Request request) throws RequestProcessorException {
        // LOG.info("Prep>>> cxid = " + request.cxid + " type = " +
        // request.type + " id = 0x" + Long.toHexString(request.sessionId));
        request.setHdr(null);
        request.setTxn(null);

        if (!request.isThrottled()) {
          pRequestHelper(request);
        }

        request.zxid = zks.getZxid();
        long timeFinishedPrepare = Time.currentElapsedTime();
        ServerMetrics.getMetrics().PREP_PROCESS_TIME.add(timeFinishedPrepare - request.prepStartTime);
        nextProcessor.processRequest(request);
        ServerMetrics.getMetrics().PROPOSAL_PROCESS_TIME.add(Time.currentElapsedTime() - timeFinishedPrepare);
    }

这里的pRequestHelper()函数是一个大switch-case，来很对不同的请求执行不同的处理，例如create请求，会执行pRequest2Txn()的逻辑：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* PrepRequestProcessor.java
	* 794行
	*/
	private void pRequestHelper(Request request) throws RequestProcessorException {
        try {
            switch (request.type) {
            case OpCode.createContainer:
            case OpCode.create:
            case OpCode.create2:
                CreateRequest create2Request = new CreateRequest();
                pRequest2Txn(request.type, zks.getNextZxid(), request, create2Request, true);
                break;
            case OpCode.createTTL:
            //...
            //case...
}

在pRequest2Txn()，主要是执行预处理阶段的主要逻辑，这里举一个比较短的例子setData请求：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* PrepRequestProcessor.java
	* 320行
	*/
	protected void pRequest2Txn(int type, long zxid, Request request, Record record, boolean deserialize) throws KeeperException, IOException, RequestProcessorException 
	{
	switch (type) {
		case OpCode.setData:
			// 检查会话是否还有效
            zks.sessionTracker.checkSession(request.sessionId, request.getOwner());
            SetDataRequest setDataRequest = (SetDataRequest) record;
            if (deserialize) {
                ByteBufferInputStream.byteBuffer2Record(request.request, setDataRequest);
            }
            // 请求的路径
            path = setDataRequest.getPath();
            validatePath(path, request.sessionId);
            // 现有数据
            nodeRecord = getRecordForPath(path);
            // 检查ACL是否有操作权限
            zks.checkACL(request.cnxn, nodeRecord.acl, ZooDefs.Perms.WRITE, request.authInfo, path, null);
            // 检查Quota是否达量
            zks.checkQuota(path, nodeRecord.data, setDataRequest.getData(), OpCode.setData);
            int newVersion = checkAndIncVersion(nodeRecord.stat.getVersion(), setDataRequest.getVersion(), path);
            // 加入处理链路
            request.setTxn(new SetDataTxn(path, setDataRequest.getData(), newVersion));
            nodeRecord = nodeRecord.duplicate(request.getHdr().getZxid());
            nodeRecord.stat.setVersion(newVersion);
            nodeRecord.stat.setMtime(request.getHdr().getTime());
            nodeRecord.stat.setMzxid(zxid);
            nodeRecord.data = setDataRequest.getData();
            nodeRecord.precalculatedDigest = precalculateDigest(
                    DigestOpCode.UPDATE, path, nodeRecord.data, nodeRecord.stat);
            setTxnDigest(request, nodeRecord.precalculatedDigest);
            addChangeRecord(nodeRecord);
            break;
	}
}

处理后，会回到pRequest()中，将request交给下一个处理器nextProcessor：ProposalRequestProcessor（集群模式下）。

事物处理器

ProposalRequestProcessor是继PrepRequestProcessor后，责任链模式上的第二个处理器。
实现类是ProposalRequestProcessor，在ProposalRequestProcessor处理器阶段，其内部又分成了三个子流程，分别是：Sync流程、Proposal流程、Commit流程。

需要首先解释：observer和follower都被称为为learner：

This class is the superclass of two of the three main actors in a ZK ensemble: Followers and Observers.

ProposalRequestProcessor的从处理主要在initialize()方法和processRequest()方法中。

SyncRequestProcessor和AckRequestProcessor链

SyncRequestProcessor

在ProposalRequestProcessor中，initialize()方法中起了一个syncProcessor的线程：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* ProposalRequestProcessor.java
	* 64行
	*/
	public void initialize() {
        syncProcessor.start();
    }

在这个类型为SyncRequestProcessor的线程中，主要处理的是将请求持久化到磁盘中：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* SyncRequestProcessor.java
	* 156行
	*/
	@Override
    public void run() {
    	//...
    	while (true) {
    		// ...
            // 不断从queuedRequests中获取request
        	Request si = queuedRequests.poll(pollTime, TimeUnit.MILLISECONDS);
        	// ...
        	if (!si.isThrottled() && zks.getZKDatabase().append(si)) {
        		//...
        		new ZooKeeperThread("Snapshot Thread") {
                                public void run() {
                                    try {
                                    	// 数据落盘
                                        zks.takeSnapshot();
                                    } catch (Exception e) {
                                        LOG.warn("Unexpected exception", e);
                                    } finally {
                                        snapThreadMutex.release();
                                    }
                                }
                            }.start();
                 //...
        	} else if (toFlush.isEmpty()) {
        		//...
                    // optimization for read heavy workloads
                    // iff this is a read or a throttled request(which doesn't need to be written to the disk),
                    // and there are no pending flushes (writes), then just pass this to the next processor
                    if (nextProcessor != null) {
                    	// 交给下一个processor：AckRequestProcessor
                        nextProcessor.processRequest(si);
                        if (nextProcessor instanceof Flushable) {
                            ((Flushable) nextProcessor).flush();
                        }
                    }
                    continue;
                }
    }

在这里，将request交给了下一个AckRequestProcessor。

AckRequestProcessor

AckRequestProcessor实现的功能是：在同意Leader的Proposal之后，给Leader回复一个ACK：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorm
	* AckRequestProcessor.java
	* 43行
	*/
	public void processRequest(Request request) {
        QuorumPeer self = leader.self;
        if (self != null) {
            request.logLatency(ServerMetrics.getMetrics().PROPOSAL_ACK_CREATION_LATENCY);
            leader.processAck(self.getId(), request.zxid, null);
        } else {
            LOG.error("Null QuorumPeer");
        }
    }

ProposalRequestProcessor、CommitProcessor链

ProposalRequestProcessor

在之前的预处理器中，调用了nextProcesser.processRequest()，将请求交给了ProposalRequestProcessor：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* ProposalRequestProcessor.java
	* 68行
	*/
	public void processRequest(Request request) throws RequestProcessorException {
        /* In the following IF-THEN-ELSE block, we process syncs on the leader.
         * If the sync is coming from a follower, then the follower
         * handler adds it to syncHandler. Otherwise, if it is a client of
         * the leader that issued the sync command, then syncHandler won't
         * contain the handler. In this case, we add it to syncHandler, and
         * call processRequest on the next processor.
         */
         // 如果请求是LearnerSyncRequest类型，也就是请求是来自于follower和observer的转发
        if (request instanceof LearnerSyncRequest) {
        	// 向Follower发送这个request
            zks.getLeader().processSync((LearnerSyncRequest) request);
            // 如果请求来自leader的客户端
        } else {
            if (shouldForwardToNextProcessor(request)) {
            	// 发给下一个处理器：CommitProcessor
                nextProcessor.processRequest(request);
            }
            if (request.getHdr() != null) {
                // We need to sync and get consensus on any transactions
                try {
                    zks.getLeader().propose(request);
                } catch (XidRolloverException e) {
                    throw new RequestProcessorException(e.getMessage(), e);
                }
                syncProcessor.processRequest(request);
            }
        }
    }

CommitProcessor

CommitProcessor用于将来到的commit request和本地已提交的request进行对比。
CommitProcessor是一个多线程的，线程之间使用队列、automatic、wait/notifyAll进行通信和协同。
CommitProcessor相当于一个gateway，决定是否继续当前的流水线操作。同一时刻可以有多个读请求和一个写请求被放通。
首先在processRequest()中，将request加入队列：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* CommitProcessor.java
	* 603行
	*/
	@Override
    public void processRequest(Request request) {
        if (stopped) {
            return;
        }
        LOG.debug("Processing request:: {}", request);
        request.commitProcQueueStartTime = Time.currentElapsedTime();
        queuedRequests.add(request);
        // If the request will block, add it to the queue of blocking requests
        if (needCommit(request)) {
            queuedWriteRequests.add(request);
            numWriteQueuedRequests.incrementAndGet();
        } else {
            numReadQueuedRequests.incrementAndGet();
        }
        wakeup();
    }

在线程中，不断从队列中取得request，对其进行处理，相关代码在org.apache.zookeeper.server.quorum.CommitProcessor.java 196-421行，代码就不放了，大致逻辑是：

• 主线程：主要是进行线程分发，主要逻辑为不停获取request队列的request，并根据sessionId分发到子线程上，同一个SessionId会被分发到同一个工作线程上；
• 工作线程：CommitWorkRequest类（继承自WorkerService.WorkRequest），核心是ArrayList<ExecutorService>，将request发到下一个processor（如果是leader，会送到Leader.ToBeAppliedRequestProcessor，follower和observer会送到FinalRequestProcessor）

最终处理器

Leader.ToBeAppliedRequestProcessor和FinalRequestProcessor链

Leader.ToBeAppliedRequestProcessor

ToBeAppliedRequestProcessor主要是维护toBeApplied列表，下一个处理器是FinalRequestProcessor。只要Leader会执行这个processor。
处理逻辑非常简单：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* Leader.java
	* 1100行
	*/
	public void processRequest(Request request) throws RequestProcessorException {
            next.processRequest(request);
            // The only requests that should be on toBeApplied are write
            // requests, for which we will have a hdr. We can't simply use
            // request.zxid here because that is set on read requests to equal
            // the zxid of the last write op.
            if (request.getHdr() != null) {
                long zxid = request.getHdr().getZxid();
                Iterator<Proposal> iter = leader.toBeApplied.iterator();
                if (iter.hasNext()) {
                    Proposal p = iter.next();
                    if (p.request != null && p.request.zxid == zxid) {
                        iter.remove();
                        return;
                    }
                }
                LOG.error("Committed request not found on toBeApplied: {}", request);
            }
        }

可以看出，只有写请求才会触发对toBeApplied列表的维护，toBeApplied
的数据结构是ConcurrentLinkedQueue<Proposal>

FinalRequestProcessor

FinalRequestProcessor是最后一个处理器（It is always at the end of a RequestProcessor chain）。
FinalRequestProcessor处理器：

• 检查outstandingChanges队列中请求的有效性，若发现这些请求已经落后于当前正在处理的请求，那么直接从outstandingChanges队列中移除
• 将事务实际提交。之前之前的请求处理仅仅将事务请求记录到了事务日志中，而内存数据库中的状态尚未改变，因此，需要将事务变更应用到内存数据库
• 回复客户端

主要逻辑代码主要在org.apache.zookeeper.server.FinalRequestProcessor.java 的147-643行

不同角色的请求链

在上面可以看出，很显然，集群中的leader、follower、observer都会有自己的请求链来执行不同的处理逻辑。单机模式的ZooKeeper也会有自己单独的请求链。
在不同的Server中的setupRequestProcessors()方法来定义好自己的请求链

Leader的请求链

Leader的请求链在LeaderZooKeeperServer中定义：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* LeaderZooKeeperServer.java
	* 65行
	*/
	@Override
    protected void setupRequestProcessors() {
        RequestProcessor finalProcessor = new FinalRequestProcessor(this);
        RequestProcessor toBeAppliedProcessor = new Leader.ToBeAppliedRequestProcessor(finalProcessor, getLeader());
        commitProcessor = new CommitProcessor(toBeAppliedProcessor, Long.toString(getServerId()), false, getZooKeeperServerListener());
        commitProcessor.start();
        ProposalRequestProcessor proposalProcessor = new ProposalRequestProcessor(this, commitProcessor);
        proposalProcessor.initialize();
        prepRequestProcessor = new PrepRequestProcessor(this, proposalProcessor);
        prepRequestProcessor.start();
        firstProcessor = new LeaderRequestProcessor(this, prepRequestProcessor);

        setupContainerManager();
    }

这里的LeaderRequestProcessor主要是处理直接连接在leader上的客户端的请求。

Follower的请求链

Follower的请求链在FollowerZooKeeperServer中定义：

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* FollowerZooKeeperServer.java
	* 69行
	*/
	@Override
    protected void setupRequestProcessors() {
        RequestProcessor finalProcessor = new FinalRequestProcessor(this);
        commitProcessor = new CommitProcessor(finalProcessor, Long.toString(getServerId()), true, getZooKeeperServerListener());
        commitProcessor.start();
        firstProcessor = new FollowerRequestProcessor(this, commitProcessor);
        ((FollowerRequestProcessor) firstProcessor).start();
        syncProcessor = new SyncRequestProcessor(this, new SendAckRequestProcessor(getFollower()));
        syncProcessor.start();
    }

Observer的请求链

Observer的请求链定义在ObserverZooKeeperServer

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server.quorum
	* ObserverZooKeeperServer.java
	* 87行
	*/
	@Override
    protected void setupRequestProcessors() {
        // We might consider changing the processor behaviour of
        // Observers to, for example, remove the disk sync requirements.
        // Currently, they behave almost exactly the same as followers.
        RequestProcessor finalProcessor = new FinalRequestProcessor(this);
        commitProcessor = new CommitProcessor(finalProcessor, Long.toString(getServerId()), true, getZooKeeperServerListener());
        commitProcessor.start();
        firstProcessor = new ObserverRequestProcessor(this, commitProcessor);
        ((ObserverRequestProcessor) firstProcessor).start();

        /*
         * Observer should write to disk, so that the it won't request
         * too old txn from the leader which may lead to getting an entire
         * snapshot.
         *
         * However, this may degrade performance as it has to write to disk
         * and do periodic snapshot which may double the memory requirements
         */
        if (syncRequestProcessorEnabled) {
            syncProcessor = new SyncRequestProcessor(this, null);
            syncProcessor.start();
        }
    }

这里的FollowerRequestProcessor主要逻辑是将request提交给leader。

单机模式请求链

单机模式请求链比较简单，定义在org.apache.zookeeper.server.ZooKeeperServer.java

	/**
	* 版本：ZooKeeper 3.8.0
	* org.apache.zookeeper.server
	* ZooKeeperServer.java
	* 746行
	*/
    protected void setupRequestProcessors() {
        RequestProcessor finalProcessor = new FinalRequestProcessor(this);
        RequestProcessor syncProcessor = new SyncRequestProcessor(this, finalProcessor);
        ((SyncRequestProcessor) syncProcessor).start();
        firstProcessor = new PrepRequestProcessor(this, syncProcessor);
        ((PrepRequestProcessor) firstProcessor).start();
    }

参考

ZooKeeper源码解析(9)-几种RequestProcessor
Zookeeper系列（二十四）Zookeeper原理解析之处理流程
Zookeeper(4) - 会话管理和事务处理
12 服务端是如何处理一次会话请求的？