使用netty 4.1.x 手写一个简单的websocket服务端

1.netty服务创建

public NettyWebsocketServer(LogWebsocketServerConfig config) {
        this.config = config;
        if (useEpoll()) {
            bossGroup = new EpollEventLoopGroup(1,
                    new EventLoopThreadFactory(1, "NettyWsEpollEventAcceptor"));
            workGroup = new EpollEventLoopGroup(config.getWorkCount(),
                    new EventLoopThreadFactory(config.getWorkCount(), "NettyWsEpollEventSelector"));
        } else {
            bossGroup = new NioEventLoopGroup(1,
                    new EventLoopThreadFactory(1, "NettyWsServerNioAcceptor"));
            workGroup = new NioEventLoopGroup(config.getWorkCount(),
                    new EventLoopThreadFactory(config.getWorkCount(), "NettyWsServerNioEventSelector"));
        }
    }

    public void createNettyServer() throws Exception {
        ServerBootstrap bootstrap = new ServerBootstrap();
        bootstrap.group(bossGroup, workGroup)
                .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
                //自定义处理器见2
                .childHandler(new SimpleServerChannelInitializer(config, bootstrap))
                .option(ChannelOption.SO_BACKLOG, 1024)
                .option(ChannelOption.SO_REUSEADDR, true)
                .childOption(ChannelOption.TCP_NODELAY, true);
        if (useEpoll()) {
            bootstrap.option(EpollChannelOption.TCP_FASTOPEN, 50);
            bootstrap.childOption(EpollChannelOption.TCP_QUICKACK, true);
            bootstrap.option(EpollChannelOption.EPOLL_MODE, EpollMode.EDGE_TRIGGERED);
        }
        ChannelFuture f = bootstrap.bind(config.getPort()).sync();
        f.addListener(v -> {
            if (!v.isSuccess()) {
                throw new RuntimeException("netty server start failed");
            } else {
                log.info("netty server started, connect port:{}, is use epoll: {}", config.getPort(), useEpoll());
            }
        });
        //开启关闭主从线程池的钩子
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            bossGroup.shutdownGracefully().syncUninterruptibly();
            workGroup.shutdownGracefully().syncUninterruptibly();
        }));
    }
    
    //是否启用native epoll
    private boolean useEpoll() {
        return config.getUseEpollNativeSelector() && NettyServerUtil.isLinuxPlatform() && Epoll.isAvailable();
    }

    /**
     * 自定义线程工厂，保证线程名称和别的业务区分
     */
    public class EventLoopThreadFactory implements ThreadFactory {

        private final AtomicInteger threadIndex = new AtomicInteger(0);

        private final int threadTotal;

        private final String threadNamePrefix;

        public EventLoopThreadFactory(int threadTotal, String threadNamePrefix) {
            this.threadTotal = threadTotal;
            this.threadNamePrefix = threadNamePrefix;
        }

        @Override
        public Thread newThread(Runnable r) {
            return new Thread(r, String.format("%s-%d-%d", threadNamePrefix, threadTotal, this.threadIndex.incrementAndGet()));
        }
    }

为了考虑实时推送的性能这里做了几点优化：

1.如果系统满足要求，则使用了netty的native epoll，开启了epoll的ET模式，而java的nio使用的LT模式。为了避免出现问题，这里除了判断系统是否支持，还增加了额外是否开启epoll的开关

2.tcp参数调优：tcp参数很多，但是java nio很多tcp参数并未暴露。所以这里在使用native epoll时增加tcp的fastopen和qucikack

2.自定义处理器

2.1 连接初始化器SimpleServerChannelInitializer

封装了创建链接相关操作,这里只摘了部分主要代码:

@ChannelHandler.Sharable
public class SimpleServerChannelInitializer extends ChannelInitializer<SocketChannel> {

...

 protected void initChannel(SocketChannel socketChannel) {
        ChannelPipeline pipeline = socketChannel.pipeline();
        //netty 自带的http解码器
        pipeline.addLast(new HttpServerCodec(config.getMaxInitialLineLength(), config.getMaxHeaderSize(),
                config.getMaxChunkSize()));
        //http聚合器
        pipeline.addLast(new HttpObjectAggregator(config.getMaxContextLength()));
        pipeline.addLast(new ChunkedWriteHandler());
        //压缩协议
        pipeline.addLast(new WebSocketServerCompressionHandler());
        //http处理器 用来握手和执行进一步操作
        pipeline.addLast(new NettyWebsocketHttpHandler(config, listener));
    }


...

}

这里有两个优化：

1.启用了压缩和大报文的聚合处理器，因为一般来说websocket推送都是大报文，可以提高网络通信的性能

2.使用了@ChannelHandler.Sharble 注解保证部分处理器是单例的，防止高并发情况下创建过多处理器对象

注意点：

需要控制maxContextLength，避免服务器遭受大报文网络攻击

2.2 http请求处理器NettyWebsocketHttpHandler

主要处理的是websocket的握手操作，在握手成功后增加自定义处理器

url后跟的clientId参数为鉴权操作预留了入口，在后续版本中实现了jwt token认证

@ChannelHandler.Sharable
public class NettyWebsocketHttpHandler extends SimpleChannelInboundHandler<FullHttpRequest> {
...

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest req) {
        handleHandshake(req, ctx);
    }

    private void handleHandshake(FullHttpRequest req, ChannelHandlerContext ctx) {
        Channel channel = ctx.channel();
        String uri = req.uri();
        if (!channel.isActive()) {
            return;
        }
        //握手前触发自定义事件回调
        listener.onHandshakeBefore(ctx, uri);
        FullHttpResponse res;
        //从uri中获取?后面的token参数
        String clientId = ChannelAttrKey.getParamKeyOne(uri, ChannelAttrKey.CLIENT_ID);
        //鉴权相关，这里暂时只是判断了一下是否为空
        if (StringUtils.isBlank(clientId)) {
            res = new DefaultFullHttpResponse(HTTP_1_1, FORBIDDEN);
            sendHttpResponse(ctx, req, res);
            return;
        }
        // Handshake
        WebSocketServerHandshakerFactory wsFactory = new WebSocketServerHandshakerFactory
                (getWebSocketLocation(req), "", true, config.getMaxContextLength());
        WebSocketServerHandshaker handshaker = wsFactory.newHandshaker(req);
        if (handshaker == null) {
            WebSocketServerHandshakerFactory.sendUnsupportedVersionResponse(channel);
        } else {
            ChannelPipeline pipeline = ctx.pipeline();
            pipeline.remove(ctx.name());
            handshaker.handshake(channel, req).addListener(future -> {
                if (future.isSuccess()) {
                    //握手成功触发自定义事件回调
                    listener.onHandshakeSuccess(ctx, uri);
                    //增加业务处理器
                    pipeline.addLast(new NettyWebsocketBusinessHandler(listener));
                    //增加心跳检测处理器
                    pipeline.addLast(new IdleStateHandler(0, 0, config.getMaxIdleTime()));
                    pipeline.addLast(new SimpleChannelIdleHandler(listener));
                } else {
                    //握手失败触发自定义事件回调
                    listener.onHandshakeFailed(ctx, uri);
                }
            });
        }
    }

 //发送http消息
 private static void sendHttpResponse(ChannelHandlerContext ctx, FullHttpRequest req, FullHttpResponse res) {
        int statusCode = res.status().code();
        if (statusCode != OK.code() && res.content().readableBytes() == 0) {
            ByteBuf buf = Unpooled.copiedBuffer(res.status().toString(), CharsetUtil.UTF_8);
            res.content().writeBytes(buf);
            buf.release();
        }
        HttpUtil.setContentLength(res, res.content().readableBytes());

        ChannelFuture f = ctx.channel().writeAndFlush(res);
        if (!HttpUtil.isKeepAlive(req) || statusCode != 200) {
            f.addListener(ChannelFutureListener.CLOSE);
        }
    }

  private static String getWebSocketLocation(FullHttpRequest req) {
        String location = req.headers().get(HttpHeaderNames.HOST) + req.uri();
        return "ws://" + location;
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) {
        ...
    }

    @Override
    public void channelInactive(ChannelHandlerContext ctx) {
        ...
    }


    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        //关闭上下文
        ...
    }
...
}

握手成功后，会增加一些参数的保存，以及创建websocket会话

    ...

    public void onHandshakeSuccess(ChannelHandlerContext ctx, String uri) {
        Channel channel = ctx.channel();
        log.info("[handshakeComplete event]channel id->{},uri->{},remote ip->{}......", channel.id(), uri,
                NettyServerUtil.getRemoteIpByChannel(channel));
        WebsocketSession session = new WebsocketSession(channel);
        channel.attr(ChannelAttrKey.SESSION_KEY).set(session);
        Optional<String> optional = Arrays.stream(uri.split("\\?")).findFirst();
        //截取?之前的参数
        if (optional.isPresent()) {
            channel.attr(ChannelAttrKey.PATH_KEY).set(optional.get());
        } else {
            channel.attr(ChannelAttrKey.PATH_KEY).set(uri);
        }

    }

    ...

websocketsession 封装了一些发送操作:

public class WebsocketSession {

    private final Channel channel;

    public WebsocketSession(Channel channel) {
        this.channel = channel;
    }

    ...


  public ChannelFuture sendText(String message) {
        return channel.writeAndFlush(new TextWebSocketFrame(message));
    }


  public ChannelFuture sendBinary(byte[] bytes) {
        ByteBuf buffer = channel.alloc().buffer(bytes.length);
        return channel.writeAndFlush(new 
            BinaryWebSocketFrame(buffer.writeBytes(bytes)));
    }

    ...
    
}

2.3 心跳检测处理器SimpleChannelIdleHandle

执行心跳检测逻辑，如果超过一定时间链接都处于空闲状态，会直接关闭上下文：

@Slf4j
@ChannelHandler.Sharable
public class SimpleChannelIdleHandler extends ChannelDuplexHandler {
    ...


    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) {
        if (evt instanceof IdleStateEvent) {
            Channel channel = ctx.channel();
            IdleStateEvent stateEvent = (IdleStateEvent) evt;
            if (stateEvent.state() == IdleState.ALL_IDLE) {
                log.warn("[ChannelIdleState event] idle channel will be closed, id->{},remote ip->{}",
                        channel.id(), NettyServerUtil.getRemoteIpByChannel(channel));
                ctx.close();
            }
        }
    }

    ....
}

2.4 业务处理器NettyWebsocketBusinessHandler

这里仅仅对不同的数据类型进行了归类，然后触发对应事件：

@Slf4j
@ChannelHandler.Sharable
public class LogWebsocketBusinessHandler extends SimpleChannelInboundHandler<WebSocketFrame> {
   
...

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, WebSocketFrame frame) {
        Channel channel = ctx.channel();
        WebsocketSession session = channel.attr(ChannelAttrKey.SESSION_KEY).get();
        ByteBuf byteBuf = frame.content();
        byte[] bytes = new byte[byteBuf.readableBytes()];
        byteBuf.readBytes(bytes);
        if (frame instanceof BinaryWebSocketFrame) {
            listener.onMessageBinary(session, bytes);
        } else if (frame instanceof TextWebSocketFrame) {
            listener.onMessageText(session, new String(bytes));
        } else if (frame instanceof PongWebSocketFrame) {
            listener.onMessagePong(session, bytes);
        } else if (frame instanceof PingWebSocketFrame) {
            listener.onMessagePing(session, bytes);
        } else if (frame instanceof ContinuationWebSocketFrame) {
            listener.onMessageContinuation(session, bytes);
        } else if (frame instanceof CloseWebSocketFrame) {
            if(channel.isActive()){
                ctx.close();
            }
        }
    }

...
}

具体的事件处理器是基于了spring的@Service自定义了处理器注解，为了实现类似于@RestController的路径逻辑：

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Service
public @interface NettyWebsocketHandlePath {

    /**
     * bean名称
     *
     * @return
     */
    @AliasFor(annotation = Service.class)
    String value() default "";

    /**
     * 路径
     * @return
     */
    String path();
}

这里获取所以加了注解的spring bean 并加入了自己的缓存，相当于路由到对应路径的处理器上处理具体业务逻辑

...

 /**
     * 获取对应bean
     */
    private void setStrategyMap() {
        //从spring获取自定义注解的相关bean
        Map<String, Object> beanMap = applicationContext.getBeansWithAnnotation(NettyWebsocketHandlePath.class);
        beanMap.forEach((k, v) -> {
            //必须是IRouteHandleStrategy接口的实现类(策略模式)
            if (v instanceof IRouteHandleStrategy) {
                IRouteHandleStrategy strategy = (IRouteHandleStrategy) v;
                NettyWebsocketHandlePath path = AnnotationUtils.findAnnotation(strategy.getClass(), NettyWebsocketHandlePath.class);
                //把bean加入自己的缓存
                STRATEGY_MAP.putIfAbsent(path.path(), strategy);
            }
        });
    }

...

    @Override
    public void onMessageBinary(WebsocketSession session, byte[] message) {
        String path = session.getAttribute(ChannelAttrKey.PATH_KEY.name());
        IRouteHandleStrategy strategy = STRATEGY_MAP.get(path);
        if (strategy != null) {
            strategy.onMessageBinary(session, message);
        } else {
            log.warn("cannot find route handler strategy for {}, please check!!!", path);
        }
    }

...

自此使用netty 4.1.x 手写一个简单的websocket服务端就结束了。