客户端
可以浏览上一篇文章https://blog.csdn.net/Confused_Tom_Cat/article/details/129413258
服务端
前置知识
- ServerCnxnFactory的实现类有NIOServerCnxnFactory（默认）和NettyServerCnxnFactory两种，本文中展示的为NettyServerCnxnFactory(与客户端进行通信，绑定的端口为2181？)
- Leader选举完成后，Leader会建立与从节点的BIO通道，来完成数据同步，对应的处理类为LearnerHandler
- Leader处理消息为责任链模式，在LeaderZooKeeperServer的setupRequestProcessors方法中设置，顺序为LeaderRequestProcessor–>PrepRequestProcessor–>ProposalRequestProcessor–>CommitProcessor–> Leader.ToBeAppliedRequestProcessor–>FinalRequestProcessor
Leader与Client之间的消息处理(展示的代码只有我认为重要的代码，不是全部)
Leader
- ConcurrentMap<Long, Proposal> outstandingProposals: 保存发送给follower的消息
- ConcurrentLinkedQueue toBeApplied ：
- Map<Long, List> pendingSyncs：保存的follower的同步请求，请求需要加（this）类锁
public Proposal propose(Request request) throws XidRolloverException {
byte[] data = SerializeUtils.serializeRequest(request);
proposalStats.setLastBufferSize(data.length);
QuorumPacket pp = new QuorumPacket(Leader.PROPOSAL, request.zxid, data, null);
Proposal p = new Proposal();
p.packet = pp;
p.request = request;
synchronized (this) {
p.addQuorumVerifier(self.getQuorumVerifier());
if (request.getHdr().getType() == OpCode.reconfig) {
self.setLastSeenQuorumVerifier(request.qv, true);
}
if (self.getQuorumVerifier().getVersion() < self.getLastSeenQuorumVerifier().getVersion()) {
p.addQuorumVerifier(self.getLastSeenQuorumVerifier());
}
lastProposed = p.packet.getZxid();
outstandingProposals.put(lastProposed, p);
sendPacket(pp);
}
ServerMetrics.getMetrics().PROPOSAL_COUNT.add(1);
return p;
}
// 发送消息给follower
void sendPacket(QuorumPacket qp) {
synchronized (forwardingFollowers) {
for (LearnerHandler f : forwardingFollowers) {
f.queuePacket(qp);
}
}
}
// 记录Leader收到的针对某个特定消息的确认
public synchronized void processAck(long sid, long zxid, SocketAddress followerAddr) {
Proposal p = outstandingProposals.get(zxid);
p.addAck(sid);
boolean hasCommitted = tryToCommit(p, zxid, followerAddr);
if (hasCommitted && p.request != null && p.request.getHdr().getType() == OpCode.reconfig) {
long curZxid = zxid;
while (allowedToCommit && hasCommitted && p != null) {
curZxid++;
p = outstandingProposals.get(curZxid);
if (p != null) {
hasCommitted = tryToCommit(p, curZxid, null);
}
}
}
}
public synchronized boolean tryToCommit(Proposal p, long zxid, SocketAddress followerAddr) {
// 确保操作是顺序提交的
if (outstandingProposals.containsKey(zxid - 1)) {
return false;
}
// 检查是否得到过半确认
if (!p.hasAllQuorums()) {
return false;
}
outstandingProposals.remove(zxid);
if (p.request != null) {
toBeApplied.add(p);
}
// 给follower发送commit消息
commit(zxid);
inform(p);
// 添加消息到commitProcessor的队列
zk.commitProcessor.commit(p.request);
if (pendingSyncs.containsKey(zxid)) {
for (LearnerSyncRequest r : pendingSyncs.remove(zxid)) {
sendSync(r);
}
}
return true;
}
NettyServerCnxnFactory
初始化方法, 绑定了处理类为NettyServerCnxnFactory.CnxnChannelHandler
NettyServerCnxnFactory() {
......
setOutstandingHandshakeLimit(Integer.getInteger(OUTSTANDING_HANDSHAKE_LIMIT, -1));
EventLoopGroup bossGroup = NettyUtils.newNioOrEpollEventLoopGroup(NettyUtils.getClientReachableLocalInetAddressCount());
EventLoopGroup workerGroup = NettyUtils.newNioOrEpollEventLoopGroup();
ServerBootstrap bootstrap = new ServerBootstrap().group(bossGroup, workerGroup)
.channel(NettyUtils.nioOrEpollServerSocketChannel())
// parent channel options
.option(ChannelOption.SO_REUSEADDR, true)
// child channels options
.childOption(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.SO_LINGER, -1)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (advancedFlowControlEnabled) {
pipeline.addLast(readIssuedTrackingHandler);
}
if (secure) {
initSSL(pipeline, false);
} else if (shouldUsePortUnification) {
initSSL(pipeline, true);
}
pipeline.addLast("servercnxnfactory", channelHandler);
}
});
this.bootstrap = configureBootstrapAllocator(bootstrap);
this.bootstrap.validate();
}
NettyServerCnxnFactory.CnxnChannelHandler
channelActive：控制服务端连接的总数量和单个IP的连接最大数，设置channel对应的NettyServerCnxn等
public void channelActive(ChannelHandlerContext ctx){
.......
final Channel channel = ctx.channel();
// 判断服务端连接的总数量
if (limitTotalNumberOfCnxns()) {
ServerMetrics.getMetrics().CONNECTION_REJECTED.add(1);
channel.close();
return;
}
// 判断单个IP的连接数
InetAddress addr = ((InetSocketAddress) channel.remoteAddress()).getAddress();
if (maxClientCnxns > 0 && getClientCnxnCount(addr) >= maxClientCnxns) {
ServerMetrics.getMetrics().CONNECTION_REJECTED.add(1);
LOG.warn("Too many connections from {} - max is {}", addr, maxClientCnxns);
channel.close();
return;
}
// 设置channel对应的NettyServerCnxn，处理channelRead得到的消息
NettyServerCnxn cnxn = new NettyServerCnxn(channel, zkServer, NettyServerCnxnFactory.this);
ctx.channel().attr(CONNECTION_ATTRIBUTE).set(cnxn);
......
}
channelRead：处理收到的的消息
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
try{
NettyServerCnxn cnxn = ctx.channel().attr(CONNECTION_ATTRIBUTE).get();
......
cnxn.processMessage((ByteBuf) msg);
......
}fianlly{
ReferenceCountUtil.release(msg);
}
}
NettyServerCnxn
在CnxnChannelHandler的channelActive方法中可以知道每个channel都对应一个NettyServerCnxn
void processMessage(ByteBuf buf) {
......
// 这里的if和else逻辑最后都调用了receiveMessage，直接关注这个方法就行
if (queuedBuffer != null) {
appendToQueuedBuffer(buf.retainedDuplicate());
processQueuedBuffer();
} else {
receiveMessage(buf);
// Have to check !closingChannel, because an error in
// receiveMessage() could have led to close() being called.
if (!closingChannel && buf.isReadable()) {
if (queuedBuffer == null) {
queuedBuffer = channel.alloc().compositeBuffer();
}
appendToQueuedBuffer(buf.retainedSlice(buf.readerIndex(), buf.readableBytes()));
}
}
}
private void receiveMessage(ByteBuf message) {
while (message.isReadable() && !throttled.get()) {
......
if (initialized) {
// 处理客户端请求
zks.processPacket(this, bb);
} else {
// 处理连接请求
zks.processConnectRequest(this, bb);
initialized = true;
}
......
}
}
ZooKeeperServer
- private final AtomicLong hzxid：服务端ID生成
public void processPacket(ServerCnxn cnxn, ByteBuffer incomingBuffer){
// 解析请求的header
InputStream bais = new ByteBufferInputStream(incomingBuffer);
BinaryInputArchive bia = BinaryInputArchive.getArchive(bais);
RequestHeader h = new RequestHeader();
h.deserialize(bia, "header");
......
Request si = new Request(cnxn, cnxn.getSessionId(), h.getXid(), h.getType(), incomingBuffer, cnxn.getAuthInfo());
si.setOwner(ServerCnxn.me);
// 下面这个方法最后调用了RequestThrottler的submitRequest方法
submitRequest(si);
......
}
public void submitRequestNow(Request si) {
......
firstProcessor.processRequest(si);
......
}
RequestThrottler
处理收到的消息的线程，收到消息后先放到队列中，然后从队列中取出并执行，有两个重要的属性
- LinkedBlockingQueue submittedRequests：存放需要处理的消息
- ZooKeeperServer zks：当前节点为Leader时，为LeaderZooKeeperServer
// 将消息放到队列中
public void submitRequest(Request request) {
request.requestThrottleQueueTime = Time.currentElapsedTime();
submittedRequests.add(request);
}
public void run() {
......
while (true) {
......
Request request = submittedRequests.take();
......
if (request != null) {
......
// 这里对应的是LeaderZooKeeperServer的submitRequestNow方法
zks.submitRequestNow(request);
}
}
......
}
LeaderRequestProcessor
public void processRequest(Request request) throws RequestProcessorException {
......
// nextProcessor为PrepRequestProcessor
nextProcessor.processRequest(request);
}
PrepRequestProcessor
继承自ZooKeeperCriticalThread，设置了消息的服务端ID
- LinkedBlockingQueue submittedRequests：保存当前需要处理的消息
- RequestProcessor nextProcessor：下一个处理类，Leader节点里为ProposalRequestProcessor
// 将需要处理的消息加入submittedRequests队列中
public void processRequest(Request request) {
request.prepQueueStartTime = Time.currentElapsedTime();
submittedRequests.add(request);
ServerMetrics.getMetrics().PREP_PROCESSOR_QUEUED.add(1);
}
// 处理submittedRequests队列中的消息
public void run() {
while (true) {
......
Request request = submittedRequests.take();
......
pRequest(request);
}
}
protected void pRequest(Request request) throws RequestProcessorException {
request.setHdr(null);
request.setTxn(null);
// request.isThrottled()默认为false
if (!request.isThrottled()) {
// 重新设置hdr和txn
pRequestHelper(request);
}
// 设置事务ID
request.zxid = zks.getZxid();
nextProcessor.processRequest(request);
}
ProposalRequestProcessor
在声明RequestProcessor时，会调用这个初始化和initialize方法
public ProposalRequestProcessor(LeaderZooKeeperServer zks, RequestProcessor nextProcessor) {
this.zks = zks;
this.nextProcessor = nextProcessor;
AckRequestProcessor ackProcessor = new AckRequestProcessor(zks.getLeader());
syncProcessor = new SyncRequestProcessor(zks, ackProcessor);
// 默认为false
forwardLearnerRequestsToCommitProcessorDisabled = Boolean.getBoolean(
FORWARD_LEARNER_REQUESTS_TO_COMMIT_PROCESSOR_DISABLED);
}
public void initialize() {
syncProcessor.start();
}
public void processRequest(Request request) throws RequestProcessorException {
//如果是LearnerSyncRequest，表明是leader做为server提供给客户端服务，并且接受到客户端的sync请求。
//如果不是是LearnerSyncRequest，则认为是需要进行投票决策，所以将request发送给leader，接着会发送给所有的follower进行投票。注意：请求可能来自于follower转发给leader的写请求，也可能是leader收到client的写请求
if (request instanceof LearnerSyncRequest) {
zks.getLeader().processSync((LearnerSyncRequest) request);
} else {
// 默认的情况下shouldForwardToNextProcessor(request)的返回值为true
if (shouldForwardToNextProcessor(request)) {
nextProcessor.processRequest(request);
}
if (request.getHdr() != null) {
try {
// 将request发送给follower
zks.getLeader().propose(request);
} catch (XidRolloverException e) {
throw new RequestProcessorException(e.getMessage(), e);
}
syncProcessor.processRequest(request);
}
}
}
CommitProcessor
继承自ZooKeeperCriticalThread，实现了RequestProcessor接口，需要同时处理等待收到ack命令的消息和等待commit的消息
- LinkedBlockingQueue committedRequests：准备提交的消息的缓存队列
- LinkedBlockingQueue queuedRequests：需要处理的消息队列
- static volatile int maxReadBatchSize：单次处理queuedRequests中的最大消息数，默认为-1，即不做数量限制且有需要commit的消息时，优先commit的消息
- LinkedBlockingQueue queuedWriteRequests：需要对数据做修改的消息队列
- static volatile int maxCommitBatchSize：单次处理queuedWriteRequests中的最大消息数，默认为1
- Map<Long, Deque> pendingRequests：如果消息为write消息或者read但之前存在未完成的write消息时，需要将消息加入到这个队列中，直到之前的write消息处理完成再从队列中取出并处理
- WorkerService workerPool：线程池
public void commit(Request request) {
committedRequests.add(request);
wakeup();
}
public void processRequest(Request request) {
request.commitProcQueueStartTime = Time.currentElapsedTime();
queuedRequests.add(request);
// If the request will block, add it to the queue of blocking requests
// 增删改等操作needCommit返回true
if (needCommit(request)) {
queuedWriteRequests.add(request);
numWriteQueuedRequests.incrementAndGet();
} else {
numReadQueuedRequests.incrementAndGet();
}
// 使用synchronized修饰的方法，调用了notifyAll();
wakeup();
}
public void run() {
try {
commitIsWaiting = !committedRequests.isEmpty();
requestsToProcess = queuedRequests.size();
// 这里因为需要同时处理两个阻塞队列中的消息，需要使用wait方法，不能使用阻塞队列自带的等待和唤醒方法
if (requestsToProcess == 0 && !commitIsWaiting) {
synchronized (this) {
while (!stopped && requestsToProcess == 0 && !commitIsWaiting) {
wait();
commitIsWaiting = !committedRequests.isEmpty();
requestsToProcess = queuedRequests.size();
}
}
}
// 分批次处理queuedRequests和committedRequests队列中的消息，先处理queuedRequests中的，
// 直到全部处理完成或者处理到单次处理的最大数量maxReadBatchSize后再处理
// committedRequests中的消息，同样最多只能处理maxCommitBatchSize的数量
// 处理queuedRequests中消息
Request request;
int readsProcessed = 0;
while (!stopped
&& requestsToProcess > 0
&& (maxReadBatchSize < 0 || readsProcessed <= maxReadBatchSize)
&& (request = queuedRequests.poll()) != null) {
requestsToProcess--;
if (needCommit(request) || pendingRequests.containsKey(request.sessionId)) {
// Add request to pending
Deque<Request> requests = pendingRequests.computeIfAbsent(request.sessionId, sid -> new ArrayDeque<>());
requests.addLast(request);
ServerMetrics.getMetrics().REQUESTS_IN_SESSION_QUEUE.add(requests.size());
} else {
readsProcessed++;
numReadQueuedRequests.decrementAndGet();
sendToNextProcessor(request);
}
// 如果没有设置maxReadBatchSize且有消息需要commit时，优先commit
if (maxReadBatchSize < 0 && !pendingRequests.isEmpty() && !committedRequests.isEmpty()) {
commitIsWaiting = true;
break;
}
}
if (commitIsWaiting && !stopped) {
// 等待workerPool中的消息全部执行完成
waitForEmptyPool();
int commitsToProcess = maxCommitBatchSize;
Set<Long> queuesToDrain = new HashSet<>();
long startWriteTime = Time.currentElapsedTime();
int commitsProcessed = 0;
while (commitIsWaiting && !stopped && commitsToProcess > 0) {
// 因为加入队列时是按照顺序加入的，所以取出的第一个时未commit中事务ID最小的
// committedRequests,queuedWriteRequests中消息的顺序是一致的，都是按照事务ID从小到大排列
request = committedRequests.peek();
if (!queuedWriteRequests.isEmpty()
&& queuedWriteRequests.peek().sessionId == request.sessionId
&& queuedWriteRequests.peek().cxid == request.cxid) {
// 检查这是否是挂起的本地写入请求，如果是，请使用提交的信息更新它。如果提交与blockedRequestQueue中排队的第一个写入相匹配，
// 我们就知道这是本地写入的提交，因为提交是按顺序接收的。否则，它必须是远程写入的提交。
Deque<Request> sessionQueue = pendingRequests.get(request.sessionId);
if (sessionQueue == null || sessionQueue.isEmpty() || !needCommit(sessionQueue.peek())) {
// 不处理这个write请求，要么这个会话中有read挂起，要么我们还没有开始write
break;
} else {
Request topPending = sessionQueue.poll();
topPending.setHdr(request.getHdr());
topPending.setTxn(request.getTxn());
topPending.setTxnDigest(request.getTxnDigest());
topPending.zxid = request.zxid;
topPending.commitRecvTime = request.commitRecvTime;
request = topPending;
numWriteQueuedRequests.decrementAndGet();
queuedWriteRequests.poll();
queuesToDrain.add(request.sessionId);
}
}
committedRequests.remove();
commitsToProcess--;
commitsProcessed++;
// 调用了Leader.ToBeAppliedRequestProcessor的processRequest方法
processWrite(request);
commitIsWaiting = !committedRequests.isEmpty();
}
readsProcessed = 0;
// 如果session在pendingRequests队列中有对应的read消息，在这里按顺序处理直到遇到write消息或者处理完全部消息
for (Long sessionId : queuesToDrain) {
Deque<Request> sessionQueue = pendingRequests.get(sessionId);
int readsAfterWrite = 0;
// 处理read消息，即不需要等待commit确认的消息
while (!stopped && !sessionQueue.isEmpty() && !needCommit(sessionQueue.peek())) {
numReadQueuedRequests.decrementAndGet();
sendToNextProcessor(sessionQueue.poll());
readsAfterWrite++;
}
if (sessionQueue.isEmpty()) {
pendingRequests.remove(sessionId);
}
}
} while (!stoppedMainLoop);
}
}
// 处理read消息，将消息加入到线程池中处理，最后调用了Leader.ToBeAppliedRequestProcessor的processRequest方法
private void sendToNextProcessor(Request request) {
numRequestsProcessing.incrementAndGet();
CommitWorkRequest workRequest = new CommitWorkRequest(request);
workerPool.schedule(workRequest, request.sessionId);
}
Leader.ToBeAppliedRequestProcessor
public void processRequest(Request request) throws RequestProcessorException {
next.processRequest(request);
// The only requests that should be on toBeApplied are write
// requests, for which we will have a hdr. We can't simply use
// request.zxid here because that is set on read requests to equal
// the zxid of the last write op.
if (request.getHdr() != null) {
long zxid = request.getHdr().getZxid();
Iterator<Proposal> iter = leader.toBeApplied.iterator();
if (iter.hasNext()) {
Proposal p = iter.next();
if (p.request != null && p.request.zxid == zxid) {
iter.remove();
return;
}
}
}
}
FinalRequestProcessor
// 将数据更新到内存中，并给客户端响应
public void processRequest(Request request) {
......
// 直选了create操作的对应代码
switch (request.type) {
......
case OpCode.create: {
lastOp = "CREA";
rsp = new CreateResponse(rc.path);
err = Code.get(rc.err);
// 将相关结果更新到内存
requestPathMetricsCollector.registerRequest(request.type, rc.path);
break;
}
......
}
// 生成响应头
ReplyHeader hdr = new ReplyHeader(request.cxid, lastZxid, err.intValue());
updateStats(request, lastOp, lastZxid);
......
// cnxn为消息对应的channel绑定的NettyServerCnxn，将响应信息发送给客户端
// 为write对应的操作，还有getData等的操作的代码没展示
responseSize = cnxn.sendResponse(hdr, rsp, "response");
......
}
SyncRequestProcessor
继承自ZooKeeperCriticalThread，实现了RequestProcessor接口
- BlockingQueue queuedRequests：需要处理的消息
- Semaphore snapThreadMutex = new Semaphore(1)：控制生成快照的并发数为1
- RequestProcessor nextProcessor：AckRequestProcessor
- Queue toFlush：已写入并正在等待刷新到磁盘的事务。基本上，这是在flush成功返回后将调用其回调的SyncItems列表。
- long lastFlushTime：上次刷新时间
// 将需要处理的消息发到阻塞队列中
public void processRequest(final Request request) {
......
queuedRequests.add(request);
}
// 将消息写到磁盘，在flush方法中调用AckRequestProcessor的processRequest方法
public void run() {
try {
while (true) {
Request si = queuedRequests.poll(pollTime, TimeUnit.MILLISECONDS);
if (si == null) {
/* We timed out looking for more writes to batch, go ahead and flush immediately */
flush();
si = queuedRequests.take();
}
if (si == REQUEST_OF_DEATH) {
break;
}
long startProcessTime = Time.currentElapsedTime();
ServerMetrics.getMetrics().SYNC_PROCESSOR_QUEUE_TIME.add(startProcessTime - si.syncQueueStartTime);
// track the number of records written to the log
if (!si.isThrottled() && zks.getZKDatabase().append(si)) {
if (shouldSnapshot()) {
resetSnapshotStats();
// roll the log
zks.getZKDatabase().rollLog();
// take a snapshot
if (!snapThreadMutex.tryAcquire()) {
LOG.warn("Too busy to snap, skipping");
} else {
new ZooKeeperThread("Snapshot Thread") {
public void run() {
try {
zks.takeSnapshot();
} catch (Exception e) {
LOG.warn("Unexpected exception", e);
} finally {
snapThreadMutex.release();
}
}
}.start();
}
}
} else if (toFlush.isEmpty()) {
// optimization for read heavy workloads
// iff this is a read or a throttled request(which doesn't need to be written to the disk),
// and there are no pending flushes (writes), then just pass this to the next processor
if (nextProcessor != null) {
nextProcessor.processRequest(si);
if (nextProcessor instanceof Flushable) {
((Flushable) nextProcessor).flush();
}
}
continue;
}
toFlush.add(si);
if (shouldFlush()) {
flush();
}
ServerMetrics.getMetrics().SYNC_PROCESS_TIME.add(Time.currentElapsedTime() - startProcessTime);
}
} catch (Throwable t) {
handleException(this.getName(), t);
}
LOG.info("SyncRequestProcessor exited!");
}
AckRequestProcessor
// 将请求作为ACK转发
public void processRequest(Request request) {
QuorumPeer self = leader.self;
if (self != null) {
leader.processAck(self.getId(), request.zxid, null);
}
}
Leader与Follower之间的消息处理
LearnerHandler
每个Learner都有一个对应的LearnerHandler，这个Learner的所有消息都由对应的LearnerHandler处理，对应的代码如下
- LinkedBlockingQueue queuedPackets：需要发送给Learner的消息
// Leader提供了客户端通信，zookeeper节点通信，节点选举三哥端口，这里是处理节点通信端口连接建立的方法
private void acceptConnections() throws IOException {
socket = serverSocket.accept();
BufferedInputStream is = new BufferedInputStream(socket.getInputStream());
LearnerHandler fh = new LearnerHandler(socket, is, Leader.this);
fh.start();
}
LearnerHandler线程的run方法，处理从相同作用的端口传来的数据包并对其进行处理，还将监听新连接。
public void run() {
......
// 开启一个新线程，发送queuedPackets中的消息
startSendingPackets();
......
// 等待收到半数以上的具有选票的节点的响应选举完成的ack，即等待选举完成
learnerMaster.waitForNewLeaderAck(getSid(), qp.getZxid());
......
while (true) {
qp = new QuorumPacket();
ia.readRecord(qp, "packet");
switch (qp.getType()) {
case Leader.ACK:
// learnerMaster即Leader类
learnerMaster.processAck(this.sid, qp.getZxid(), sock.getLocalSocketAddress());
break;
......
}
}
}
// 启动线程，将queuedPackets队列中的数据包转发到Follower
protected void startSendingPackets() {
if (!sendingThreadStarted) {
// Start sending packets
new Thread() {
public void run() {
Thread.currentThread().setName("Sender-" + sock.getRemoteSocketAddress());
try {
sendPackets();
} catch (InterruptedException e) {
LOG.warn("Unexpected interruption", e);
}
}
}.start();
sendingThreadStarted = true;
}
}
Follower的消息处理
选举完成后，如果状态为FOLLOWING，会调用创建Follower角色的如下代码
setFollower(makeFollower(logFactory));
follower.followLeader();
makeFollower方法创建follower，并绑定了FollowerZooKeeperServer；followLeader中启动了follower的相关逻辑，包括与Leader建立连接，数据同步，消息处理等
Follower
Leader与单个Follower的过程，数字表示顺序，黄色方块为线程，绿色方块为队列
启动Follower
void followLeader() throws InterruptedException {
// 建立与Leader的连接，如果设置了异步发送，会建立LearnerSender线程来异步发送消息
connectToLeader(leaderServer.addr, leaderServer.hostname);
......
// 从Leader同步消息,并调用FollowerZooKeeperServer的setupRequestProcessors方法初始化Follower的消息处理链
// 第一阶段消息的处理链(PROPOSAL)：SyncRequestProcessor-->SendAckRequestProcessor
// 第二阶段消息的处理链（COMMIT）：FollowerRequestProcessor-->CommitProcessor-->FinalRequestProcessor
syncWithLeader(newEpochZxid);
......
QuorumPacket qp = new QuorumPacket();
while (this.isRunning()) {
readPacket(qp);
processPacket(qp);
}
}
// 处理Leader发来的消息，这里只展示PROPOSAL和COMMIT两种类型的消息
protected void processPacket(QuorumPacket qp) throws Exception {
switch (qp.getType()) {
case Leader.PROPOSAL:
fzk.logRequest(hdr, txn, digest);
case Leader.COMMIT:
fzk.commit(qp.getZxid());
}
}
FollowerZooKeeperServer
- LinkedBlockingQueue pendingTxns：保存收到的PROPOSAL类型的消息，收到对应的COMMIT消息时，从队列中删除该消息
- PROPOSAL消息处理链：SyncRequestProcessor–>SendAckRequestProcessor
- COMMIT消息处理链：FollowerRequestProcessor–>CommitProcessor–>FinalRequestProcessor
Follower处理PROPOSAL类型的消息，将消息加入到SyncRequestProcessor队列中后，从队列中捞出写入文件后给Leader发送ACK消息
public void logRequest(TxnHeader hdr, Record txn, TxnDigest digest) {
Request request = new Request(hdr.getClientId(), hdr.getCxid(), hdr.getType(), hdr, txn, hdr.getZxid());
request.setTxnDigest(digest);
if ((request.zxid & 0xffffffffL) != 0) {
pendingTxns.add(request);
}
// 由SyncRequestProcessor处理，写到文件中，然后调用SendAckRequestProcessor发送ack给Leader
syncProcessor.processRequest(request);
}
Follower处理COMMIT类型的消息，将消息加入到CommitProcessor队列中后，从队列中捞出写入内存后给Leader发送ACK消息
public void commit(long zxid) {
if (pendingTxns.size() == 0) {
return;
}
long firstElementZxid = pendingTxns.element().zxid;
Request request = pendingTxns.remove();
commitProcessor.commit(request);
}
791
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
