1.前言
rocketmq的各个组件之间的通信都是用rocketmq-remoting这个组件实现，而这个组件底层采用的是netty来进行通信的，所以学习这个组件我们可以学到netty的使用。
2.netty基本介绍
在介绍这个组件之前，我们来了解一下什么是netty。
2.1 NIO
2.1.1 传统AIO的不足
传统AIO是，每次请求到来的时候，会启动一个线程处理请求，并且每个线程在该请求处理完成后，才会处理下一个请求，所以AIO也叫同步阻塞IO，当请求量大的时候，会创建大量请求。
2.2.2 NIO
NIO是同步非阻塞IO，他会启动一个selector线程来接收请求，当没有请求的时候selector会阻塞。如果有读写请求的时候，会交给线程池来处理，这就是同步非阻塞IO。
2.2.3 举例
下面是nio的server的一个例子：
public class SimpleServer {
public static void main(String[] args) throws IOException {
Logger logger = LoggerFactory.getLogger(SimpleServer.class);
//创建服务端channel
ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
//设置channel非阻塞
serverSocketChannel.configureBlocking(false);
//获得selector
Selector selector = Selector.open();
//把channel注册到selector上,现在还没有给key设置感兴趣的事件
SelectionKey selectionKey = serverSocketChannel.register(selector, 0, serverSocketChannel);
//给key设置感兴趣的事件
selectionKey.interestOps(SelectionKey.OP_ACCEPT);
//绑定端口号
serverSocketChannel.bind(new InetSocketAddress(8080));
//然后开始接受连接,处理事件,整个处理都在一个死循环之中
while (true) {
//当没有事件到来的时候，这里是阻塞的,有事件的时候会自动运行
selector.select();
//如果有事件到来，这里可以得到注册到该selector上的所有的key，每一个key上都有一个channel
Set<SelectionKey> selectionKeys = selector.selectedKeys();
//得到集合的迭代器
Iterator<SelectionKey> keyIterator = selectionKeys.iterator();
while (keyIterator.hasNext()) {
//得到每一个key
SelectionKey key = keyIterator.next();
//首先要从集合中把key删除，否则会一直报告该key
keyIterator.remove();
//接下来就要处理事件，判断selector轮询到的是什么事件，并根据事件作出回应
//如果是连接事件
if (key.isAcceptable()) {
//得到服务端的channel,这里有两种方式获得服务端的channel，一种是直接获得,一种是通过attachment获得
//因为之前把服务端channel注册到selector上时，同时把serverSocketChannel放进去了
ServerSocketChannel channel = (ServerSocketChannel)key.channel();
//ServerSocketChannel attachment = (ServerSocketChannel)key.attachment();
//得到客户端的channel
SocketChannel socketChannel = channel.accept();
socketChannel.configureBlocking(false);
//接下来就要管理客户端的channel了，和服务端的channel的做法相同，客户端的channel也应该被注册到selector上
//通过一次次的轮询来接受并处理channel上的相关事件
//把客户端的channel注册到之前已经创建好的selector上
SelectionKey socketChannelKey = socketChannel.register(selector, 0, socketChannel);
//给客户端的channel设置可读事件
socketChannelKey.interestOps(SelectionKey.OP_READ);
logger.info("客户端连接成功！");
//连接成功之后，用客户端的channel写回一条消息
socketChannel.write(ByteBuffer.wrap("我发送成功了".getBytes()));
logger.info("向客户端发送数据成功！");
}
//如果接受到的为可读事件，说明要用客户端的channel来处理
if (key.isReadable()) {
//同样有两种方式得到客户端的channel，这里只列出一种
SocketChannel channel = (SocketChannel)key.channel();
//分配字节缓冲区来接受客户端传过来的数据
ByteBuffer buffer = ByteBuffer.allocate(1024);
//向buffer写入客户端传来的数据
int len = channel.read(buffer);
logger.info("读到的字节数：" + len);
if (len == -1) {
channel.close();
break;
}else{
//切换buffer的读模式
buffer.flip();
logger.info(Charset.defaultCharset().decode(buffer).toString());
}
}
}
}
}
}2.2 netty的线程模型
上面是netty经典的线程模型图，可以看出netty包含了两个NioEventLoop（两个线程池），分别是bossGroup和workerGroup。workerGroup主要用来封装请求，bossGroup用来处理请求，从图中可以得出Netty工作步骤如下:
1.client发送请求到达bossGroup，bossGroup将请求注册到selector上面这一动作封装成一个runnableTask，并且交给NioEventLoop里面的线程池执行。这里线程池其实是每个NioEventloopGroup包含多个NipEventLoop，每个NioEventLoop包含一个线程，所以可以理解为NioEventLoopGroup包含一个线程池。
2.selector收到请求过后，会判断请求是要读、写或者是建立连接，并且将其封装成一个runnableTask，并且交给workGorup线程池执行。workGroup线程池执行对于的IO操作。
3.worker线程池执行对于的IO操作的时候，会将请求发送给pipeline，pipeline其实就是一个拦截器链，会完成请求的编解码。我们对netty进行业务处理的时候，其实也是去实现ChannelHandler，使其成为拦截器的一个节点。
为什么Netty采用这种线程模型可以提升吞吐量？
其实思想就是将一个完整的业务链路分别拆分，然后并且交给不同的线程执行。在现实生活中，我们去吃饭的时候，假设老板只有一个人，他需要招待我们点餐->炒菜->上菜这几个步骤过后，才能去招待下一桌客人，这其实就是AIO。但实际生活中，是点餐人员来招待点餐，点好过后，就把菜单给厨师，厨师开始炒菜，这个时候点餐人员就可以招待下一桌人员了，这其实就是NIO。所以思想源于生活。
2.3 netty的pipeline
在NIO中，客户端的连接会被抽象为socketchannel，服务端的连接被抽象为serversocketchannel，而在netty里面的连接可以用channel来表示。
netty有两种处理器，分别是入站处理器和出站处理器，入站处理器，处理读消息，
3.rocketmq是如何利用Netty通信
3.1 rocketmq的核心组件
3.1.1 NettyRemotingAbstract的基本组成和作用
1.有哪些组件构成
NettyRemotingAbstract的主要作用是接收Netty的请求，根据请求找到一个响应或者请求的processor，然后调用对应的客户端实现的processor来处理结果。组成如下：
/**
* Semaphore to limit maximum number of on-going one-way requests, which protects system memory footprint.
*/
//oneway请求的信号量，通过它来控制同时发送oneway请求的并发数
protected final Semaphore semaphoreOneway;
/**
* Semaphore to limit maximum number of on-going asynchronous requests, which protects system memory footprint.
*/
//异步请求的信号量，通过它来控制同时发送异步请求的并发数
protected final Semaphore semaphoreAsync;
/**
* This map caches all on-going requests.
*/
//存储请求的opaque 处理请求的future
protected final ConcurrentMap<Integer /* opaque */, ResponseFuture> responseTable =
new ConcurrentHashMap<Integer, ResponseFuture>(256);
/**
* This container holds all processors per request code, aka, for each incoming request, we may look up the
* responding processor in this map to handle the request.
*/
//享元模式的体现，这个是存储对应的请求类型和响应处理器的一个table，key:RequestCode里面 value:一个处理器pair，其实就是真正的对消息
//做业务逻辑处理的地方，为了增大存储，采用线程池进行异步处理。NettyRequestProcessor有不同的子类，来处理不同的消息，比如DefaultRequestProcessor
//就是处理Namesrv与broker消息之间通信的processor
protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);
/**
* Executor to feed netty events to user defined {@link ChannelEventListener}.
*/
//处理Netty事件的一个线程池，比如根据Netty传入的空闲，连接事件等进行处理,比如传入一个Close事件，以Namesrve为例子，表示
//断开Namesrv和broker之间的连接，所以需要清除Namesrv里面routinfo的关于broker的信息
protected final NettyEventExecutor nettyEventExecutor = new NettyEventExecutor();
/**
* The default request processor to use in case there is no exact match in {@link #processorTable} per request code.
*/
//如果通过请求的唯一id没有找到对应processor，就用这个defaultRequestProcessor来处理请求
protected Pair<NettyRequestProcessor, ExecutorService> defaultRequestProcessor;
/**
* SSL context via which to create {@link SslHandler}.
*/
//ssl加密通信的组件
protected volatile SslContext sslContext;
/**
* custom rpc hooks
*/
//在请求处理的前后会加入一些拦截器，这里拦截器就是RPCHook
protected List<RPCHook> rpcHooks = new ArrayList<RPCHook>();
2.rocketmq是如何处理响应的 - responseTable和ResponseFuture
rocketmq每个请求会有一个唯一id，opqueueId的，并且在发送请求过后会将opaque
和一个响应理器:ResponseFuture加入到responseTable中。
public class ResponseFuture {
//请求的唯一id
private final int opaque;
//对应的channel
private final Channel processChannel;
//请求的超时时间
private final long timeoutMillis;
//在收到响应的时候，设置处理该请求的回调函数,交由调用方实现
private final InvokeCallback invokeCallback;
private final long beginTimestamp = System.currentTimeMillis();
//同步请求阻塞请求线程，等待处理结果
private final CountDownLatch countDownLatch = new CountDownLatch(1);
//做请求的并发限制
private final SemaphoreReleaseOnlyOnce once;
private final AtomicBoolean executeCallbackOnlyOnce = new AtomicBoolean(false);
//接收到的command响应
private volatile RemotingCommand responseCommand;
private volatile boolean sendRequestOK = true;
private volatile Throwable cause;
public ResponseFuture(Channel channel, int opaque, long timeoutMillis, InvokeCallback invokeCallback,
SemaphoreReleaseOnlyOnce once) {
this.opaque = opaque;
this.processChannel = channel;
this.timeoutMillis = timeoutMillis;
this.invokeCallback = invokeCallback;
this.once = once;
}
/**
* 执行回调方法
*/
public void executeInvokeCallback() {
if (invokeCallback != null) {
//通过AtomicBoolean保证回调方法只会执行一次
if (this.executeCallbackOnlyOnce.compareAndSet(false, true)) {
invokeCallback.operationComplete(this);
}
}
}
public void release() {
if (this.once != null) {
this.once.release();
}
}
public boolean isTimeout() {
long diff = System.currentTimeMillis() - this.beginTimestamp;
return diff > this.timeoutMillis;
}
//通过同步方法，阻塞线程，保证同步调用接收到请求
public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
return this.responseCommand;
}
//当同步调用的响应来时，调用putResponse方法，会通过countDownLatch的countDown方法，释放掉调用方的阻塞线程
public void putResponse(final RemotingCommand responseCommand) {
this.responseCommand = responseCommand;
this.countDownLatch.countDown();
}
//异步请求通过setResponseCommand设置响应
public void setResponseCommand(RemotingCommand responseCommand) {
this.responseCommand = responseCommand;
}
}额外补充一点，下面是SemaphoreReleaseOnlyOnce 的标准用法，因为信号量在释放的时候，可能也会有并发问题，所以需要用AtomicBoolean 来保证释放的并发安全。详情可参考：java高并发高频面试题:Sempahore的使用场景与常见误区_java semaphore坑-CSDN博客
public class SemaphoreReleaseOnlyOnce {
private final AtomicBoolean released = new AtomicBoolean(false);
private final Semaphore semaphore;
public SemaphoreReleaseOnlyOnce(Semaphore semaphore) {
this.semaphore = semaphore;
}
public void release() {
if (this.semaphore != null) {
if (this.released.compareAndSet(false, true)) {
this.semaphore.release();
}
}
}
public Semaphore getSemaphore() {
return semaphore;
}
}3.rocketmq是如何处理请求的 - processorTable
NettyRemotingAbstract里面有一个组件processorTable，记录了请求类型和processor的映射关系，根据request code判断出是哪种processor，然后通过processor调用processRequest方法执行处理逻辑。
protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);public interface NettyRequestProcessor {
RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
throws Exception;
boolean rejectRequest();
}
真正的处理逻辑在processor的实现类里面，也就是每个调用法实现的子类。
3.1.2 NettyRemotingServer的基本组成
1.有哪些组件构成
NettyRemotingServer其实就是一个Netty的server端。
public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
// netty的启动器
private final ServerBootstrap serverBootstrap;
// 接收客户端的连接请求，并且将处理完的请求交给workGroup
private final EventLoopGroup eventLoopGroupSelector;
//netty的workGroup，负责连接过后的读写请求
private final EventLoopGroup eventLoopGroupBoss;
//netty的相关的config
private final NettyServerConfig nettyServerConfig;
//执行callback操作的线程池
private final ExecutorService publicExecutor;
//监听channel的断开或者连接事件，并且根据事件类型执行对应的操作
private final ChannelEventListener channelEventListener;
//定时任务调度器，主要用来扫描连接是否超时
private final Timer timer = new Timer("ServerHouseKeepingService", true);
//ChannelEventListener监听到各种事件后，通过各种事件交由该线程池执行
private DefaultEventExecutorGroup defaultEventExecutorGroup;
private int port = 0;
//ssl需要加入的各种handler
private static final String HANDSHAKE_HANDLER_NAME = "handshakeHandler";
private static final String TLS_HANDLER_NAME = "sslHandler";
private static final String FILE_REGION_ENCODER_NAME = "fileRegionEncoder";
// sharable handlers
//ssl需要handshake的handler
private HandshakeHandler handshakeHandler;
//编码器
private NettyEncoder encoder;
//连接处理器
private NettyConnectManageHandler connectionManageHandler;
//处理请求和响应的处理器
private NettyServerHandler serverHandler;
}接下来我们先看一下这个nettyserver是如何启动起来的。对应的就是NettyRemotingServer中的start方法：
public void start() {
//启动对连接事件的处理器
this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
nettyServerConfig.getServerWorkerThreads(),
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
}
});
//构造netty的pipeline链条中的各个handler
prepareSharableHandlers();
//构造netty的启动器
ServerBootstrap childHandler =
this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
.channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
.option(ChannelOption.SO_BACKLOG, nettyServerConfig.getServerSocketBacklog())
.option(ChannelOption.SO_REUSEADDR, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.childOption(ChannelOption.TCP_NODELAY, true)
.localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ch.pipeline()
//添加ssl需要的handShakeHandler
.addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, handshakeHandler)
.addLast(defaultEventExecutorGroup,
//rocketmq自定义通信协议解编码器
encoder,
//rocketmq自定义通信协议解码器
new NettyDecoder(),
//netty自带的空闲处理器
new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
//rocketmq的连接处理器
connectionManageHandler,
//接收到请求后真正做逻辑处理的地方
serverHandler
);
}
});
try {
//绑定端口后，启动netty
ChannelFuture sync = this.serverBootstrap.bind().sync();
InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
this.port = addr.getPort();
} catch (InterruptedException e1) {
throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
}
if (this.channelEventListener != null) {
this.nettyEventExecutor.start();
}
//启动扫描响应的定时任务
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
NettyRemotingServer.this.scanResponseTable();
} catch (Throwable e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000);
}其实现在我们已经大概清除rocketmq在收到请求后，会经过哪些handler处理了。
接下来我们来看看netty的各个组件的作用:
2 ssl连接的handshakeHandler
这个是用来判断服务端和客户端是否要通过ssl对连接进行加密，如果需要则会想netty的pipeline中加入sslHandler和fileRegionEncoder，请求在经过这借个handler过后，会自动与客户端建立ssl连接。
class HandshakeHandler extends SimpleChannelInboundHandler<ByteBuf> {
private final TlsMode tlsMode;
private static final byte HANDSHAKE_MAGIC_CODE = 0x16;
HandshakeHandler(TlsMode tlsMode) {
this.tlsMode = tlsMode;
}
@Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception {
// mark the current position so that we can peek the first byte to determine if the content is starting with
// TLS handshake
msg.markReaderIndex();
byte b = msg.getByte(0);
if (b == HANDSHAKE_MAGIC_CODE) {
switch (tlsMode) {
case DISABLED:
ctx.close();
log.warn("Clients intend to establish an SSL connection while this server is running in SSL disabled mode");
break;
case PERMISSIVE:
case ENFORCING:
if (null != sslContext) {
ctx.pipeline()
.addAfter(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, TLS_HANDLER_NAME, sslContext.newHandler(ctx.channel().alloc()))
.addAfter(defaultEventExecutorGroup, TLS_HANDLER_NAME, FILE_REGION_ENCODER_NAME, new FileRegionEncoder());
log.info("Handlers prepended to channel pipeline to establish SSL connection");
} else {
ctx.close();
log.error("Trying to establish an SSL connection but sslContext is null");
}
break;
default:
log.warn("Unknown TLS mode");
break;
}
} else if (tlsMode == TlsMode.ENFORCING) {
ctx.close();
log.warn("Clients intend to establish an insecure connection while this server is running in SSL enforcing mode");
}
// reset the reader index so that handshake negotiation may proceed as normal.
msg.resetReaderIndex();
try {
// Remove this handler
ctx.pipeline().remove(this);
} catch (NoSuchElementException e) {
log.error("Error while removing HandshakeHandler", e);
}
// Hand over this message to the next .
ctx.fireChannelRead(msg.retain());
}
}3.解码用的NettyEncoder和rocketmq的协议RemotingCommand
public class RemotingCommand {
//响应或者请求码
private int code;
//采用的语言
private LanguageCode language = LanguageCode.JAVA;
private int version = 0;
//请求id，每个请求有个唯一id
private int opaque = requestId.getAndIncrement();
private int flag = 0;
//备注
private String remark;
//扩展字段信息
private HashMap<String, String> extFields;
//请求头，一般每种请求有自己对应的请求头，比如发送给namesrv删除topic的请求会放DeleteTopicRequestHeader请求头
private transient CommandCustomHeader customHeader;
//序列化方式，默认采用ROCKETMQ
private SerializeType serializeTypeCurrentRPC = serializeTypeConfigInThisServer;
//请求体
private transient byte[] body;
}其实一般框架在采用netty通信的时候，都会自定义自己的协议，比如定义一个魔数做校验，顶一个请求头传输协议信息，然后是请求体等。在编码的时候，一般采用的方式是，字段长度+字段内容的方式形成一个字节流。而在解码的时候，首先会先读取字段长度，然后根据长度获取字段的内容，完成解码。rocketmq的编解码方式也是这样处理的。
public static RemotingCommand decode(final ByteBuf byteBuffer) throws RemotingCommandException {
//读取字段长度
int length = byteBuffer.readableBytes();
int oriHeaderLen = byteBuffer.readInt();
int headerLength = getHeaderLength(oriHeaderLen);
if (headerLength > length - 4) {
throw new RemotingCommandException("decode error, bad header length: " + headerLength);
}
//读取内容
RemotingCommand cmd = headerDecode(byteBuffer, headerLength, getProtocolType(oriHeaderLen));
int bodyLength = length - 4 - headerLength;
byte[] bodyData = null;
if (bodyLength > 0) {
bodyData = new byte[bodyLength];
byteBuffer.readBytes(bodyData);
}
cmd.body = bodyData;
return cmd;
}而nettyEnocder的解码其实就是调用的就是RemotingCommand 的解码
public class NettyDecoder extends LengthFieldBasedFrameDecoder {
private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
private static final int FRAME_MAX_LENGTH =
Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));
public NettyDecoder() {
super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
}
@Override
public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
ByteBuf frame = null;
try {
frame = (ByteBuf) super.decode(ctx, in);
if (null == frame) {
return null;
}
return RemotingCommand.decode(frame);
} catch (Exception e) {
log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
RemotingUtil.closeChannel(ctx.channel());
} finally {
if (null != frame) {
frame.release();
}
}
return null;
}
}4.如何让客户端,比如Namesrv在监听到连接断开后做自己逻辑处理-NettyConnectManageHandler
比如Namesrv在监听到channel断开后需要清除broker的routeInfo信息，而broker在监听到channel断开后，也需要做自己的处理。这个是如何实现的呢？
NettyRomotingServer在Netty的channelpipeline中加入了NettyConnectManageHandler，这个组件会监听到channel所触发的事件，比如是Close事件。便会将其打包成一个NettyEvent，并且放到NettyEventExecutor中的一个LinkedBlockingQueue中，并且启动该线程池一直消费该事件，来调用注册到这里面的ChannelEventListener方法中对应的onChannelClose进行处理，每次构造一个NettyRemotingServer的时候，会传入ChannelEventListener，比如Namesrv就是BrokerHousekeepingService。
//接收channel事件，并且启动线程池来根据事件来分发,调用listener来处理时事件
class NettyEventExecutor extends ServiceThread {
//一个阻塞队列来接收channel相关事件
private final LinkedBlockingQueue<NettyEvent> eventQueue = new LinkedBlockingQueue<NettyEvent>();
private final int maxSize = 10000;
public void putNettyEvent(final NettyEvent event) {
int currentSize = this.eventQueue.size();
if (currentSize <= maxSize) {
this.eventQueue.add(event);
} else {
log.warn("event queue size [{}] over the limit [{}], so drop this event {}", currentSize, maxSize, event.toString());
}
}
@Override
public void run() {
log.info(this.getServiceName() + " service started");
final ChannelEventListener listener = NettyRemotingAbstract.this.getChannelEventListener();
while (!this.isStopped()) {
try {
NettyEvent event = this.eventQueue.poll(3000, TimeUnit.MILLISECONDS);
if (event != null && listener != null) {
switch (event.getType()) {
case IDLE:
//根据对应事件，调用对应evetentListener来做处理
listener.onChannelIdle(event.getRemoteAddr(), event.getChannel());
break;
case CLOSE:
listener.onChannelClose(event.getRemoteAddr(), event.getChannel());
break;
case CONNECT:
listener.onChannelConnect(event.getRemoteAddr(), event.getChannel());
break;
case EXCEPTION:
listener.onChannelException(event.getRemoteAddr(), event.getChannel());
break;
default:
break;
}
}
} catch (Exception e) {
log.warn(this.getServiceName() + " service has exception. ", e);
}
}
log.info(this.getServiceName() + " service end");
}
@Override
public String getServiceName() {
return NettyEventExecutor.class.getSimpleName();
}
}//获取到当前remotingServer中的eventListener
public abstract ChannelEventListener getChannelEventListener();
//将事件放入到eventExecutor中
public void putNettyEvent(final NettyEvent event) {
this.nettyEventExecutor.putNettyEvent(event);
}5.netty的基本配置-NettyServerConfig
netty的基本配置，可以提供使用方配置nettyRemotingServer
public class NettyServerConfig implements Cloneable {
//namesrv监听端口号
private int listenPort = 8888;
//netty的worker线程数量
private int serverWorkerThreads = 8;
private int serverCallbackExecutorThreads = 0;
//netty中bossGroup数量
private int serverSelectorThreads = 3;
//最多同时运行发送多少个oneWay请求
private int serverOnewaySemaphoreValue = 256;
//最多同时发送多少个异步请求
private int serverAsyncSemaphoreValue = 64;
//netty长连接最大的空闲时间
private int serverChannelMaxIdleTimeSeconds = 120;
//netty发送方的bytebuffer的大小
private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
//netty接收方的bytebuffer的大小
private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
//netty的流量控制参数，当超过高水位时，channel将不可写，直到地域低水位时，channel才会变成可写状态
private int writeBufferHighWaterMark = NettySystemConfig.writeBufferHighWaterMark;
private int writeBufferLowWaterMark = NettySystemConfig.writeBufferLowWaterMark;
private int serverSocketBacklog = NettySystemConfig.socketBacklog;
//在netty中是否开启池化，开启池化后，当bytebuffer的数据被写完过后，不会立刻释放，而达到内存重复利用
private boolean serverPooledByteBufAllocatorEnable = true;
}6.服务端如何处理器请求和连接-NettyServerHandler
class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
@Override
protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
processMessageReceived(ctx, msg);
}
}NettyServerHandler是pipeline的最后一个handler，是做逻辑处理的地方。它其实调用的就是NettyRemotingAbstract的processMessageReceived方法:
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
switch (cmd.getType()) {
case REQUEST_COMMAND:
processRequestCommand(ctx, cmd);
break;
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
可以看出它其实就是对收到的信息进行判断，看是请求还是响应，netty是一个双向的通信软件，比如broker向namesrv请求删除某个topic，namesrv收到的就是一个请求，namesrv删除后会给broker一个答复，此时broker收到的就是一个响应。
如果收到的是请求：
public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
//根据请求类型获取到请求的processor
final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
final int opaque = cmd.getOpaque();
if (pair != null) {
Runnable run = new Runnable() {
@Override
public void run() {
try {
String remoteAddr = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
//执行请求前的拦截器
doBeforeRpcHooks(remoteAddr, cmd);
//这个是该请求在服务端处理完过后，需要执行的拦截器，切实就是将opqueue和响应类型放入到response里面
final RemotingResponseCallback callback = new RemotingResponseCallback() {
@Override
public void callback(RemotingCommand response) {
doAfterRpcHooks(remoteAddr, cmd, response);
if (!cmd.isOnewayRPC()) {
if (response != null) {
response.setOpaque(opaque);
response.markResponseType();
response.setSerializeTypeCurrentRPC(cmd.getSerializeTypeCurrentRPC());
try {
ctx.writeAndFlush(response);
} catch (Throwable e) {
log.error("process request over, but response failed", e);
log.error(cmd.toString());
log.error(response.toString());
}
} else {
}
}
}
};
//异步请求执行该逻辑
if (pair.getObject1() instanceof AsyncNettyRequestProcessor) {
AsyncNettyRequestProcessor processor = (AsyncNettyRequestProcessor)pair.getObject1();
//调用客户端的processRequest方法，执行对请求的响应
processor.asyncProcessRequest(ctx, cmd, callback);
} else {
//同步请求执行该逻辑
NettyRequestProcessor processor = pair.getObject1();
RemotingCommand response = processor.processRequest(ctx, cmd);
callback.callback(response);
}
}
}
}
}
}如果收到的是响应：
public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
final int opaque = cmd.getOpaque();
//根据响应获取到发送请求时塞入的回调逻辑
final ResponseFuture responseFuture = responseTable.get(opaque);
if (responseFuture != null) {
//将响应加入到responseCommand中
responseFuture.setResponseCommand(cmd);
responseTable.remove(opaque);
if (responseFuture.getInvokeCallback() != null) {
//如果是异步请求，需要执行回调逻辑，即调用方的业务逻辑
executeInvokeCallback(responseFuture);
} else {
//同步请求，通过countDownLatch唤醒发起调用的线程
responseFuture.putResponse(cmd);
responseFuture.release();
}
} else {
log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
log.warn(cmd.toString());
}
} //其实就是异步调用，调用方在调用的时候，会塞入回调函数，当接收到服务端响应过后，利用在callBackExecutor中执行回调逻辑
private void executeInvokeCallback(final ResponseFuture responseFuture) {
boolean runInThisThread = false;
ExecutorService executor = this.getCallbackExecutor();
if (executor != null) {
try {
executor.submit(new Runnable() {
@Override
public void run() {
try {
responseFuture.executeInvokeCallback();
} catch (Throwable e) {
log.warn("execute callback in executor exception, and callback throw", e);
} finally {
responseFuture.release();
}
}
});
} catch (Exception e) {
runInThisThread = true;
log.warn("execute callback in executor exception, maybe executor busy", e);
}
} else {
runInThisThread = true;
}
if (runInThisThread) {
try {
responseFuture.executeInvokeCallback();
} catch (Throwable e) {
log.warn("executeInvokeCallback Exception", e);
} finally {
responseFuture.release();
}
}
}7.调用方如何里利用NettyRomotingServer发起调用
NettyRomotingServer提供了三种调用模式分别是同步调用、异步调用、onway调用。
同步调用:
public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
final long timeoutMillis)
throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
//获取请求的唯一id
final int opaque = request.getOpaque();
try {
//设置responseFuture，便是在响应回来的时候，交给ResponseFutre
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
//放入到responseTable中
this.responseTable.put(opaque, responseFuture);
final SocketAddress addr = channel.remoteAddress();
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
responseFuture.setSendRequestOK(true);
return;
} else {
responseFuture.setSendRequestOK(false);
}
//当响应完成过后，设置响应结果
responseTable.remove(opaque);
responseFuture.setCause(f.cause());
responseFuture.putResponse(null);
log.warn("send a request command to channel <" + addr + "> failed.");
}
});
//同步调用的核心，会阻塞当前线程，一直到结果返回
RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
if (null == responseCommand) {
if (responseFuture.isSendRequestOK()) {
throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
responseFuture.getCause());
} else {
throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
}
}
return responseCommand;
} finally {
this.responseTable.remove(opaque);
}
}异步调用:
public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
final InvokeCallback invokeCallback)
throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
long beginStartTime = System.currentTimeMillis();
final int opaque = request.getOpaque();
//异步调用会通过信号量限制请求数量，先获取到信号量
boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
if (acquired) {
final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
long costTime = System.currentTimeMillis() - beginStartTime;
//如果获取时间超过异步调用的超时时间，会释放信号量，并且抛出异常
if (timeoutMillis < costTime) {
once.release();
throw new RemotingTimeoutException("invokeAsyncImpl call timeout");
}
//设置responseFuture,这里会给responseFuture中设置等待该调用返回的超时时间，会启动一个定时线程去扫描，如果超时会抛出异常
final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
this.responseTable.put(opaque, responseFuture);
try {
//设置调用成功的回调函数
channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture f) throws Exception {
if (f.isSuccess()) {
//设置responseFuture的状态为OK
responseFuture.setSendRequestOK(true);
return;
}
requestFail(opaque);
log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
}
});
} catch (Exception e) {
responseFuture.release();
log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
}
} else {
if (timeoutMillis <= 0) {
throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
} else {
String info =
String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
timeoutMillis,
this.semaphoreAsync.getQueueLength(),
this.semaphoreAsync.availablePermits()
);
log.warn(info);
throw new RemotingTimeoutException(info);
}
}
}
oneway调用：就是发送请求后，不关心结果
8.如果调用超过了设置的超时时间，怎么处理?
nettyRemotingServer在start的时候有这样一行代码:其实就是开启一个定时任务去扫描responeTable，并且启动线程去执行reponseFuture中的回调逻辑。由于此时响应还没有回来，所以响应码并不是成功，在执行executeInvokeCallback会多错误进行处理。
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
NettyRemotingServer.this.scanResponseTable();
} catch (Throwable e) {
log.error("scanResponseTable exception", e);
}
}
}, 1000 * 3, 1000); public void scanResponseTable() {
final List<ResponseFuture> rfList = new LinkedList<ResponseFuture>();
//变量responseTable
Iterator<Entry<Integer, ResponseFuture>> it = this.responseTable.entrySet().iterator();
while (it.hasNext()) {
Entry<Integer, ResponseFuture> next = it.next();
ResponseFuture rep = next.getValue();
//计算调用开始时间+超时时间是否是超过当前时间，如果没有超时，便会从当前responseTable中移除，同时执行调用方的响应逻辑
if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
rep.release();
it.remove();
rfList.add(rep);
log.warn("remove timeout request, " + rep);
}
}
for (ResponseFuture rf : rfList) {
try {
//同时一会执行调用方的回调方法（也即调用方的响应逻辑）
executeInvokeCallback(rf);
} catch (Throwable e) {
log.warn("scanResponseTable, operationComplete Exception", e);
}
}
}3.2 调用法如何利用NettyRemotingServer进行网络连接
3.2.1 构造一个NettyRemotingServer，并且启动该服务器
//1.构NettyRemotingServer
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
3.2.2 实现processor，并且注册到remotingServer中
public class DefaultRequestProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
private static InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
protected final NamesrvController namesrvController;
protected Set<String> configBlackList = new HashSet<>();
public DefaultRequestProcessor(NamesrvController namesrvController) {
this.namesrvController = namesrvController;
initConfigBlackList();
}
}
//启动remotingServer
this.remotingServer.start(); private void registerProcessor() {
if (namesrvConfig.isClusterTest()) {
this.remotingServer.registerDefaultProcessor(new ClusterTestRequestProcessor(this, namesrvConfig.getProductEnvName()),
this.remotingExecutor);
} else {
this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.remotingExecutor);
}
}上面其实可以在processor中实现接收到请求后对请求进行处理。
3.2.3 利用remotingServer的invokeSync方法发送请求
public RemotingCommand callClient(final Channel channel,
final RemotingCommand request
) throws RemotingSendRequestException, RemotingTimeoutException, InterruptedException {
return this.brokerController.getRemotingServer().invokeSync(channel, request, 10000);
}3.3 nettyRemotingServer的执行流程
3.4 总结
nettyRemotingServer是框架中网络通信的常见的实现。主要采用netty进行网络通信。其中有很多写法也是在框架中常见的，比如在调用时，可以将调用请求封装成一个Runnable对象，通过runnable放到线程池中执行。这样做的主要目的就是为了增加吞吐量。
再比如在监听连接事件，后通过listener进行事件监听后处理，采用了享元模式和监听器模式和生产者消费者模式，将事件封装成一个NettyEvent并且加入一个阻塞队列，然后启动一个线程池去消费这个阻塞队列，并且调用监听者的listener方法。
还有就是对于请求结果的处理，其实就是在请求发送后将请求id，和处理结果的回调方法放入到一个map中，当结果返回后，直接执行回调方法即可。
可以看出Rocketmq中有大量的设计模式的应用，值的我们学习。
1293
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
