目录
前言
本篇文章将从以下几方面分析RocketMQ的通信模块：
- RocketMQ的线程模型
- RocketMQ如何实现同步、异步及单向调用
一、RocketMQ线程模型
在RocketMQ中NettyRemotingServer与NettyRemotingClient分别代表客户端与服务端，下面分别看下NettyRemotingServer与NettyRemotingClient的初始化及启动过程。
1. NettyRemotingServer初始化
NettyRemotingServer初始化的过程具体如下：
（1）初始化ServerBootstrap，用于引导服务端
（2）设置nettyServerConfig和channelEventListener，其中nettyServerConfig是服务端的通信配置
（3）初始化publicExecutor线程池
（4）判断服务端是否启用了Epoll，判断条件有三个：当前平台是否是Linux、broker端配置文件的配置项useEpollNativeSelector是否为true、当前平台是否支持Epoll
（5）如果启用了Epoll则eventLoopGroupBoss和eventLoopGroupSelector在初始化时将使用EpollEventLoopGroup，否则使用NioEventLoopGroup（eventLoopGroupBoss负责监听TCP网络连接请求事件OP_ACCEPT，建立好连接后提交注册任务到eventLoopGroupSelector，eventLoopGroupSelector会将socketChannel注册到selector，监听OP_READ事件，当网络数据可读后，提交任务到defaultEventExecutorGroup）
public NettyRemotingServer(final NettyServerConfig nettyServerConfig,
final ChannelEventListener channelEventListener) {
super(nettyServerConfig.getServerOnewaySemaphoreValue(), nettyServerConfig.getServerAsyncSemaphoreValue());
this.serverBootstrap = new ServerBootstrap();
this.nettyServerConfig = nettyServerConfig;
this.channelEventListener = channelEventListener;
int publicThreadNums = nettyServerConfig.getServerCallbackExecutorThreads();
if (publicThreadNums <= 0) {
publicThreadNums = 4;
}
this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerPublicExecutor_" + this.threadIndex.incrementAndGet());
}
});
if (useEpoll()) {
this.eventLoopGroupBoss = new EpollEventLoopGroup(1, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyEPOLLBoss_%d", this.threadIndex.incrementAndGet()));
}
});
this.eventLoopGroupSelector = new EpollEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
private int threadTotal = nettyServerConfig.getServerSelectorThreads();
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyServerEPOLLSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
}
});
} else {
this.eventLoopGroupBoss = new NioEventLoopGroup(1, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyNIOBoss_%d", this.threadIndex.incrementAndGet()));
}
});
this.eventLoopGroupSelector = new NioEventLoopGroup(nettyServerConfig.getServerSelectorThreads(), new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
private int threadTotal = nettyServerConfig.getServerSelectorThreads();
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyServerNIOSelector_%d_%d", threadTotal, this.threadIndex.incrementAndGet()));
}
});
}
loadSslContext();
}
2. NettyRemotingServer启动
NettyRemotingServer启动过程具体如下：
（1）初始化defaultEventExecutorGroup
（2）调用prepareSharableHandlers方法初始化handshakeHandler、encoder、connectionManageHandler和serverHandler，其中serverHandler是业务处理的handler
（3）通过serverBootstrap来引导服务端，这里需要注意handler的顺序以及其类型（Inbound或者 Outbound）
（4）serverBootstrap绑定端口并同步返回绑定结果
（5）如果channelEventListener不为null，则启动nettyEventExecutor线程池（channelEventListener是监测连接的）
（6）启动定时任务，定期扫描已经过期的请求
public void start() {
this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
nettyServerConfig.getServerWorkerThreads(),
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
}
});
prepareSharableHandlers();
ServerBootstrap childHandler =
this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
.channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, 1024)
.option(ChannelOption.SO_REUSEADDR, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.childOption(ChannelOption.TCP_NODELAY, true)
.childOption(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
.childOption(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
.localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline()
.addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, handshakeHandler)
.addLast(defaultEventExecutorGroup,
encoder,
new NettyDecoder(),
new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
connectionManageHandler,
serverHandler
);
}
});
if (nettyServerConfig.isServerPooledByteBufAllocatorEnable()) {
childHandler.childOption(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
}
try {
ChannelFuture sync = this.serverBootstrap.bind().sync();
InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
this.port = addr.getPort();
} catch (InterruptedException e1) {
throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
}
if (this.channelEventListener != null) {
this.nettyEventExecutor.start();
}
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
NettyRemotingServer.this.scanResponseTable();
} catch (Throwable e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000);
}
3. NettyRemotingServer线程模型
下图是服务端的线程模型，从图中可以看到RocketMQ中NettyRemotingServer遵循了reactor多线程模型并在其基础上做了优化和扩展。其中EventLoopGroupBoss为reactor主线程，负责监听TCP网络连接请求事件OP_ACCEPT，建立好连接后提交注册任务到eventLoopGroupSelector。eventLoopGroupSelector会将socketChannel注册到selector上，监听OP_READ事件，当网络数据可读后提交任务到defaultEventExecutorGroup。defaultEventExecutorGroup在收到数据后会先对数据进行SSL验证、编解码、空闲检查、网络连接管理，最后会交给NettyServerHandler进行业务上处理。在进行业务处理时会先在服务端缓存中processorTable根据请求的requestCode获取该类请求对应的processor和线程池，然后将请求封装成一个RequestTask，最后提交到线程池中执行。
补充：
这里有一个很重要的数据缓存–processorTable，其数据结构是<requestCode, Pair<processor, executor>>，那么该缓存中的数据是在哪里添加的呢？这其实与broker的启动有关，在broker启动过程中会对BrokerController进行初始化，在其初始化的过程中与processorTable相关的两点是：
（1）初始化各类业务处理的线程池
（2）调用registerProcessor方法将请求的code、处理该请求的processor以及线程池注册到broker端的remotingServer和fastRemotingServer中
这样就完成了processorTable中数据的添加，此外还需要注意一点就是remotingServer和fastRemotingServer都是NettyRemotingServer的实例，其区别在于两个实例的端口是不一样的，remotingServer的端口是10911，fastRemotingServer的端口是10909，那么问题来了：fastRemotingServer和remotingServer除端口外其余网络配置都一样，那么为什么还要再多一个实例呢，官方的回答是隔离读写操作，具体见：https://github.com/apache/rocketmq/issues/1510
最后就是pipeline中的handler是可以进行动态调整的。
if (result) {
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.clientHousekeepingService);
NettyServerConfig fastConfig = (NettyServerConfig) this.nettyServerConfig.clone();
fastConfig.setListenPort(nettyServerConfig.getListenPort() - 2);
this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);
this.sendMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getSendMessageThreadPoolNums(),
this.brokerConfig.getSendMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.sendThreadPoolQueue,
new ThreadFactoryImpl("SendMessageThread_"));
this.pullMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getPullMessageThreadPoolNums(),
this.brokerConfig.getPullMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.pullThreadPoolQueue,
new ThreadFactoryImpl("PullMessageThread_"));
this.replyMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
this.brokerConfig.getProcessReplyMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.replyThreadPoolQueue,
new ThreadFactoryImpl("ProcessReplyMessageThread_"));
this.queryMessageExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getQueryMessageThreadPoolNums(),
this.brokerConfig.getQueryMessageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.queryThreadPoolQueue,
new ThreadFactoryImpl("QueryMessageThread_"));
this.adminBrokerExecutor =
Executors.newFixedThreadPool(this.brokerConfig.getAdminBrokerThreadPoolNums(), new ThreadFactoryImpl(
"AdminBrokerThread_"));
this.clientManageExecutor = new ThreadPoolExecutor(
this.brokerConfig.getClientManageThreadPoolNums(),
this.brokerConfig.getClientManageThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.clientManagerThreadPoolQueue,
new ThreadFactoryImpl("ClientManageThread_"));
this.heartbeatExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getHeartbeatThreadPoolNums(),
this.brokerConfig.getHeartbeatThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.heartbeatThreadPoolQueue,
new ThreadFactoryImpl("HeartbeatThread_", true));
this.endTransactionExecutor = new BrokerFixedThreadPoolExecutor(
this.brokerConfig.getEndTransactionThreadPoolNums(),
this.brokerConfig.getEndTransactionThreadPoolNums(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.endTransactionThreadPoolQueue,
new ThreadFactoryImpl("EndTransactionThread_"));
this.consumerManageExecutor =
Executors.newFixedThreadPool(this.brokerConfig.getConsumerManageThreadPoolNums(), new ThreadFactoryImpl(
"ConsumerManageThread_"));
this.registerProcessor();
4. NettyRemotingClient初始化
NettyRemotingClient初始化的过程具体如下：
（1）初始化两个信号量，其中semaphoreOneway线程并发数是256、semaphoreAsync线程并发数是64，semaphoreOneway是用来控制单向调用并发量，semaphoreAsync是用来控制异步调用并发量
（2）设置nettyClientConfig和channelEventListener，其中nettyClientConfig是客户端端的通信配置
（3）初始化publicExecutor线程池
（4）初始化eventLoopGroupWorker
（5）判断客户端是否启用了TLS，如果启用了则构建sslContext
public NettyRemotingClient(final NettyClientConfig nettyClientConfig,
final ChannelEventListener channelEventListener) {
super(nettyClientConfig.getClientOnewaySemaphoreValue(), nettyClientConfig.getClientAsyncSemaphoreValue());
this.nettyClientConfig = nettyClientConfig;
this.channelEventListener = channelEventListener;
int publicThreadNums = nettyClientConfig.getClientCallbackExecutorThreads();
if (publicThreadNums <= 0) {
publicThreadNums = 4;
}
this.publicExecutor = Executors.newFixedThreadPool(publicThreadNums, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyClientPublicExecutor_" + this.threadIndex.incrementAndGet());
}
});
this.eventLoopGroupWorker = new NioEventLoopGroup(1, new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, String.format("NettyClientSelector_%d", this.threadIndex.incrementAndGet()));
}
});
if (nettyClientConfig.isUseTLS()) {
try {
sslContext = TlsHelper.buildSslContext(true);
log.info("SSL enabled for client");
} catch (IOException e) {
log.error("Failed to create SSLContext", e);
} catch (CertificateException e) {
log.error("Failed to create SSLContext", e);
throw new RuntimeException("Failed to create SSLContext", e);
}
}
}
5. NettyRemotingClient启动
NettyRemotingClient的启动过程与NettyRemotingServer类似，具体如下：
（1）初始化defaultEventExecutorGroup
（2）通过bootstrap来引导客户端，这里同样需要注意handler的顺序及其类型（Inbound或者 Outbound），NettyClientHandler是客户端业务处理的handler
（3）启动定时任务，定期扫描已经过期的请求
（4）如果channelEventListener不为空则启动nettyEventExecutor线程池
public void start() {
this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
nettyClientConfig.getClientWorkerThreads(),
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
}
});
Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
.option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
.option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
.handler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (nettyClientConfig.isUseTLS()) {
if (null != sslContext) {
pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
log.info("Prepend SSL handler");
} else {
log.warn("Connections are insecure as SSLContext is null!");
}
}
pipeline.addLast(
defaultEventExecutorGroup,
new NettyEncoder(),
new NettyDecoder(),
new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
new NettyConnectManageHandler(),
new NettyClientHandler());
}
});
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
NettyRemotingClient.this.scanResponseTable();
} catch (Throwable e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000);
if (this.channelEventListener != null) {
this.nettyEventExecutor.start();
}
}
6. NettyRemotingClient线程模型
NettyRemotingClient的线程模型与NettyRemotingServer类似，这里就不过多阐述了。
二、RocketMQ如何实现同步、异步及单向调用
1.同步调用
同步调用的实现如下：
（1）分别启动客户端和服务端
（2）客户端调用invokeSync方法发起同步调用，在该方法中会调用getAndCreateChannel方法来建立与服务端的连接，创建连接的核心是在createChannel方法，具体如下：
private Channel createChannel(final String addr) throws InterruptedException {
ChannelWrapper cw = this.channelTables.get(addr);
if (cw != null && cw.isOK()) {
return cw.getChannel();
}
if (this.lockChannelTables.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
boolean createNewConnection;
cw = this.channelTables.get(addr);
if (cw != null) {
if (cw.isOK()) {
return cw.getChannel();
} else if (!cw.getChannelFuture().isDone()) {
createNewConnection = false;
} else {
this.channelTables.remove(addr);
createNewConnection = true;
}
} else {
createNewConnection = true;
}
if (createNewConnection) {
ChannelFuture channelFuture = this.bootstrap.connect(RemotingHelper.string2SocketAddress(addr));
log.info("createChannel: begin to connect remote host[{}] asynchronously", addr);
cw = new ChannelWrapper(channelFuture);
this.channelTables.put(addr, cw);
}
} catch (Exception e) {
log.error("createChannel: create channel exception", e);
} finally {
this.lockChannelTables.unlock();
}
} else {
log.warn("createChannel: try to lock channel table, but timeout, {}ms", LOCK_TIMEOUT_MILLIS);
}
if (cw != null) {
ChannelFuture channelFuture = cw.getChannelFuture();
if (channelFuture.awaitUninterruptibly(this.nettyClientConfig.getConnectTimeoutMillis())) {
if (cw.isOK()) {
log.info("createChannel: connect remote host[{}] success, {}", addr, channelFuture.toString());
return cw.getChannel();
} else {
log.warn("createChannel: connect remote host[" + addr + "] failed, " + channelFuture.toString(), channelFuture.cause());
}
} else {
log.warn("createChannel: connect remote host[{}] timeout {}ms, {}", addr, this.nettyClientConfig.getConnectTimeoutMillis(),
channelFuture.toString());
}
}
return null;
}
（3）在invokeSync方法中同步调用的核心实现在invokeSyncImpl方法中，在该方法中重要的操作如下：
- 将本次请求的opaque和responseFuture存储在responseTable中，其中opaque相当于requestId，在同一个连接上的不同请求标识码，与响应消息中的相对应，responseFuture中记录了opaque、请求的连接channel、响应responseCommand、超时时间、回调函数和信号量（在同步调用中回调函数即信号量都为空，在异步调用中回调函数和信号量都不为空，在单向调用中回调函数为空但是信号量不为空）
- 通过channel的writeAndFlush方法将request发送给服务端
- 通过对responseFuture执行waitResponse方法来阻塞当前线程知道收到响应（这里是使用CountDownLatch来实现的）
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
final int opaque = request.getOpaque();
try {
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
responseFuture.putResponse(null);
log.warn("send a request command to channel <" + addr + "> failed.");
}
});
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}
public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
return this.responseCommand;
}
这里还有一个小的细节需要注意：在invokeSyncImpl方法中将请求发送给服务端时添加了一个监听器，这里如果发送成功则会将responseFuture的sendRequestOK属性设置为true否则设置为false，在客户端线程被唤醒后会对响应为空的情况进行进一步分析，如果responseFuture的sendRequestOK属性为true则表示发送成功请求成功但是没有收到响应此时会抛出超时异常；如果responseFuture的sendRequestOK属性为false则表示发送请求失败此时会抛出发送失败异常，所以可以看出发送失败和超时是两种异常。
（4）服务端的eventLoopGroupSelector在监听到该请求可读后会将其提交给defaultEventExcutorGroup，之后defaultEventExcutorGroup会按照服务端启动过程中添加的handler来对请求进行处理，例如SSL验证、解码数据、空闲检查、网络连接管理以及最终的业务处理（NettyServerHandler）。这里从NettyServerHandler来看下broker端是如何处理请求：NettyServerHandler重写了channelRead0方法，该方法调用了processMessageReceived方法，在这个方法中按照RemotingCommand的类型进行了区分，分别是REQUEST_COMMAND和RESPONSE_COMMAND，这里收到的是REQUEST_COMMAND所以详细看下processRequestCommand方法，在该方法中会先从processorTable中根据请求的requestCode获取该类型请求对应的processor和线程池，然后构建一个RequestTask，最后提交给该类型请求对应的线程池中进行处理，这里注意下RequestTask中会将请求提交给对应processor的来处理，此外还需要注意一点：在processRequestCommand方法中在封装任务时有一个callback，在这个callback中对请求的类型进行了判断，如果不是单向调用，即同步调用或者异步调用，则在响应中设置好收到的请求的opaque并将响应返回给客户端。
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
switch (cmd.getType()) {
case REQUEST_COMMAND:
processRequestCommand(ctx, cmd);
break;
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
final int opaque = cmd.getOpaque();
if (pair != null) {
Runnable run = new Runnable() {
@Override
public void run() {
try {
doBeforeRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd);
final RemotingResponseCallback callback = new RemotingResponseCallback() {
@Override
public void callback(RemotingCommand response) {
doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd, response);
if (!cmd.isOnewayRPC()) {
if (response != null) {
response.setOpaque(opaque);
response.markResponseType();
try {
ctx.writeAndFlush(response);
} catch (Throwable e) {
log.error("process request over, but response failed", e);
log.error(cmd.toString());
log.error(response.toString());
}
} else {
}
}
}
};
if (pair.getObject1() instanceof AsyncNettyRequestProcessor) {
AsyncNettyRequestProcessor processor = (AsyncNettyRequestProcessor)pair.getObject1();
processor.asyncProcessRequest(ctx, cmd, callback);
} else {
NettyRequestProcessor processor = pair.getObject1();
RemotingCommand response = processor.processRequest(ctx, cmd);
callback.callback(response);
}
} catch (Throwable e) {
log.error("process request exception", e);
log.error(cmd.toString());
if (!cmd.isOnewayRPC()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_ERROR,
RemotingHelper.exceptionSimpleDesc(e));
response.setOpaque(opaque);
ctx.writeAndFlush(response);
}
}
}
};
if (pair.getObject1().rejectRequest()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
"[REJECTREQUEST]system busy, start flow control for a while");
response.setOpaque(opaque);
ctx.writeAndFlush(response);
return;
}
try {
final RequestTask requestTask = new RequestTask(run, ctx.channel(), cmd);
pair.getObject2().submit(requestTask);
} catch (RejectedExecutionException e) {
if ((System.currentTimeMillis() % 10000) == 0) {
log.warn(RemotingHelper.parseChannelRemoteAddr(ctx.channel())
+ ", too many requests and system thread pool busy, RejectedExecutionException "
+ pair.getObject2().toString()
+ " request code: " + cmd.getCode());
}
if (!cmd.isOnewayRPC()) {
final RemotingCommand response = RemotingCommand.createResponseCommand(RemotingSysResponseCode.SYSTEM_BUSY,
"[OVERLOAD]system busy, start flow control for a while");
response.setOpaque(opaque);
ctx.writeAndFlush(response);
}
}
} else {
String error = " request type " + cmd.getCode() + " not supported";
final RemotingCommand response =
RemotingCommand.createResponseCommand(RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED, error);
response.setOpaque(opaque);
ctx.writeAndFlush(response);
log.error(RemotingHelper.parseChannelRemoteAddr(ctx.channel()) + error);
}
}
（5）服务端处理完请求后会调用ChannelHandlerContext.writeAndFlush从当前handler依次调用所有的出站handler最终将响应返回给客户端
（6）客户端在收到服务端的响应后最终是在NettyClientHandler中处理的，NettyClientHandler重写了channelRead0方法，同样在processMessageReceived方法中对RemotingCommand的类型进行了区分，这里由于是RESPONSE_COMMAND，所以会执行processResponseCommand方法，该方法首先会根据响应中的opaque从responseTable中获取该请求对应的响应结果responseFuture，然后将响应结果设置到responseFuture，然后从responseTable中删除该记录。最后会判断responseFuture中是否有回调函数，由于是同步调用所以没有回调函数，此时会对responseFuture中的countDownLatch执行countDown操作进而唤醒被阻塞的线程并返回responseFuture中的responseCommand
public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
final int opaque = cmd.getOpaque();
final ResponseFuture responseFuture = responseTable.get(opaque);
if (responseFuture != null) {
responseFuture.setResponseCommand(cmd);
responseTable.remove(opaque);
if (responseFuture.getInvokeCallback() != null) {
executeInvokeCallback(responseFuture);
} else {
responseFuture.putResponse(cmd);
responseFuture.release();
}
} else {
log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
log.warn(cmd.toString());
}
}
最后用一张图来总结同步调用的实现过程，具体如下：
2.异步调用
异步调用的实现如下：
（1）分别启动客户端和服务端
（2）客户端调用invokeAsync方法发起异步调用，在该方法中会调用getAndCreateChannel方法来建立与服务端的连接，创建连接的核心是在createChannel方法与同步调用是一样的
（3）在invokeAsync方法中同步调用的核心实现在invokeAsyncImpl方法中，在该方法中重要的操作如下：
- 从semaphoreAsync中获取信号量，该信号量用来控制异步调用并发量
- 与同步调用一样，构建responseFuture对象并将opaque与responseFuture缓存在responseTable中，与同步调用不一样的是responseFuture对象中会包含回调函数及获取的信号量
- 通过channel的writeAndFlush方法将请求发送给服务端
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
final InvokeCallback invokeCallback)
throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
long beginStartTime = System.currentTimeMillis();
final int opaque = request.getOpaque();
boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
if (acquired) {
final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
long costTime = System.currentTimeMillis() - beginStartTime;
if (timeoutMillis < costTime) {
once.release();
throw new RemotingTimeoutException("invokeAsyncImpl call timeout");
}
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
this.responseTable.put(opaque, responseFuture);
try {
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
}
requestFail(opaque);
log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
}
});
} catch (Exception e) {
responseFuture.release();
log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
}
} else {
if (timeoutMillis <= 0) {
throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
} else {
String info =
String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
timeoutMillis,
this.semaphoreAsync.getQueueLength(),
this.semaphoreAsync.availablePermits()
);
log.warn(info);
throw new RemotingTimeoutException(info);
}
}
}
（4）服务端收到请求后的处理流程与同步调用是一样的，这里不再过多阐述，最终服务端将请求发送给客户端，这里需要值得注意的是：在processRequestCommand方法中在封装任务时有一个callback，在这个callback中对请求的类型进行了判断，如果不是单向调用，即同步调用或者异步调用，则在响应中设置好收到的请求的opaque并将响应返回给客户端
（5）客户端在收到服务端的响应后最终是在NettyClientHandler中处理的，NettyClientHandler重写了channelRead0方法，同样在processMessageReceived方法中对RemotingCommand的类型进行了区分，这里由于是RESPONSE_COMMAND，所以会执行processResponseCommand方法，该方法首先会根据响应中的opaque从responseTable中获取该请求对应的响应结果responseFuture，然后将响应结果设置到responseFuture，然后从responseTable中删除该记录。最后会判断responseFuture中是否有回调函数，由于是异步调用所以是有回调函数的，最后会执行executeInvokeCallback方法，该方法会将回调函数封装成一个任务并将其提交到publicExecutor线程池中执行，执行完回调函数后会释放responseFuture，具体就是将持有的信号量释放
private void executeInvokeCallback(final ResponseFuture responseFuture) {
boolean runInThisThread = false;
ExecutorService executor = this.getCallbackExecutor();
if (executor != null) {
try {
executor.submit(new Runnable() {
@Override
public void run() {
try {
responseFuture.executeInvokeCallback();
} catch (Throwable e) {
log.warn("execute callback in executor exception, and callback throw", e);
} finally {
responseFuture.release();
}
}
});
} catch (Exception e) {
runInThisThread = true;
log.warn("execute callback in executor exception, maybe executor busy", e);
}
} else {
runInThisThread = true;
}
if (runInThisThread) {
try {
responseFuture.executeInvokeCallback();
} catch (Throwable e) {
log.warn("executeInvokeCallback Exception", e);
} finally {
responseFuture.release();
}
}
}
最后用一张图来总结异步调用的实现过程，具体如下：
3.单向调用
单向调用的实现如下：
（1）分别启动客户端和服务端
（2）客户端调用invokeOneway方法发起异步调用，在该方法中会调用getAndCreateChannel方法来建立与服务端的连接，创建连接的核心是在createChannel方法与同步调用是一样的
（3）在invokeOneway方法中同步调用的核心实现在invokeOnewayImpl方法中，在该方法中重要的操作如下：
- 从semaphoreOneway中获取信号量，该信号量用来控制单向调用并发量
- 与同步调用一样，构建responseFuture对象并将opaque与responseFuture缓存在responseTable中，与同步调用不一样的是responseFuture对象中会包含获取的信号量
- 通过channel的writeAndFlush方法将请求发送给服务端
- 在请求发送成功后在ChannelFutureListener中会释放信号量
（4）服务端收到请求后的处理流程与同步调用是一样的，这里不再过多阐述，但是有一点需要注意：在processRequestCommand方法中在封装任务时有一个callback，在这个callback中对请求的类型进行了判断，如果是单向调用则不会将响应返回给客户端
最后用一张图来总结单向调用的实现过程，具体如下：
395
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
