Netty实现静态服务器之坑

概述

最近在重构一个H5的项目，需要把关系型数据库干掉以及集成静态页面，尽可能减少部署成本。

原本这个项目是独立的spring boot工程前端后分离的，页面是通过nginx跑起来的。重构后需要和MG(netty实现)服务端进行整合，在经过一系列尝试以后终于将后端能力重构到netty上，并且将复杂的逻辑关系用磁盘存储方式成功避免事物问题，从而去掉了关系型数据库。

页面初加载

本以为万事大吉，数据库都去掉了，就一个简单的页面而已，然噩梦刚开始。

MG的架构采用 netty+netty-resteasy+spring实现的，如果想要加载页面就必需增加自己的handler实现静态服务器。

开始一通百度然而没有这么做的案例，netty做静态服务的例子到时有，于是开始翻netty-resteasy的源码，有没有相关的扩展点。

服务入口是从这里开始的：

Properties pro = getNettyConfig();
NettyJaxrsServer netty = new NettyJaxrsServer();
netty.setDeployment(initDeployment());
netty.setPort(Integer.parseInt(pro.getProperty("port")));
netty.setRootResourcePath("/");
netty.setSecurityDomain(null);
netty.setExecutorThreadCount(Integer.parseInt(pro.getProperty("workerCount")));
netty.setMaxRequestSize(Integer.parseInt(pro.getProperty("maxPostSize")));
netty.setBacklog(Integer.parseInt(pro.getProperty("backlogSize")));
netty.setIdleTimeout(Integer.parseInt(pro.getProperty("readTimeout")));
Map<ChannelOption, Object> channelOptions = new HashMap<ChannelOption, Object>();
channelOptions.put(ChannelOption.TCP_NODELAY, true);
channelOptions.put(ChannelOption.CONNECT_TIMEOUT_MILLIS, Integer.parseInt(pro.getProperty("connectTimeout")));
channelOptions.put(ChannelOption.SO_RCVBUF, Integer.parseInt(pro.getProperty("receiveBuffer")));
channelOptions.put(ChannelOption.SO_SNDBUF, Integer.parseInt(pro.getProperty("sendBuffer")));
channelOptions.put(ChannelOption.SO_TIMEOUT, Integer.parseInt(pro.getProperty("readTimeout")));
channelOptions.put(ChannelOption.SO_KEEPALIVE, false);
netty.setChannelOptions(channelOptions);
netty.start();

找到start方法看到这是传统的netty主从reactor模式的标准写法。

public void start() {
        eventLoopGroup = new NioEventLoopGroup(ioWorkerCount);
        eventExecutor = new NioEventLoopGroup(executorThreadCount);
        deployment.start();
        // Configure the server.
        bootstrap.group(eventLoopGroup)
                .channel(NioServerSocketChannel.class)
                .childHandler(createChannelInitializer())
                .option(ChannelOption.SO_BACKLOG, backlog)
                .childOption(ChannelOption.SO_KEEPALIVE, true);

        for (Map.Entry<ChannelOption, Object> entry : channelOptions.entrySet()) {
            bootstrap.option(entry.getKey(), entry.getValue());
        }

        for (Map.Entry<ChannelOption, Object> entry : childChannelOptions.entrySet()) {
            bootstrap.childOption(entry.getKey(), entry.getValue());
        }

        final InetSocketAddress socketAddress;
        if (null == hostname || hostname.isEmpty()) {
            socketAddress = new InetSocketAddress(configuredPort);
        } else {
            socketAddress = new InetSocketAddress(hostname, configuredPort);
        }

        Channel channel = bootstrap.bind(socketAddress).syncUninterruptibly().channel();
        runtimePort = ((InetSocketAddress) channel.localAddress()).getPort();
    }

可以看到createChannelInitializer（）这个方法其实是真正去注册accept事件的，那我们需要把我们自己的handler给到ChannelInitializer正常的话就可以实现我们的需求了。

于是继续跟踪createChannelInitializer：

private ChannelInitializer<SocketChannel> createChannelInitializer() {
        final RequestDispatcher dispatcher = createRequestDispatcher();
        if (sslContext == null && sniConfiguration == null) {
            return new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    setupHandlers(ch, dispatcher, HTTP);
                }
            };
        } else if (sniConfiguration == null) {
            return new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    SSLEngine engine = sslContext.createSSLEngine();
                    engine.setUseClientMode(false);
                    ch.pipeline().addFirst(new SslHandler(engine));
                    setupHandlers(ch, dispatcher, HTTPS);
                }
            };
        } else {
            return new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addFirst(new SniHandler(sniConfiguration.buildMapping()));
                    setupHandlers(ch, dispatcher, HTTPS);
                }
            };
        }
    }
    
    // 以上几个方法最后都会调用这个方法
     private void setupHandlers(SocketChannel ch, RequestDispatcher dispatcher, RestEasyHttpRequestDecoder.Protocol protocol) {
        ChannelPipeline channelPipeline = ch.pipeline();
        channelPipeline.addLast(channelHandlers.toArray(new ChannelHandler[channelHandlers.size()]));
        channelPipeline.addLast(new HttpRequestDecoder(maxInitialLineLength, maxHeaderSize, maxChunkSize));
        channelPipeline.addLast(new HttpResponseEncoder());
        channelPipeline.addLast(new HttpObjectAggregator(maxRequestSize));
        channelPipeline.addLast(httpChannelHandlers.toArray(new ChannelHandler[httpChannelHandlers.size()]));
        channelPipeline.addLast(new RestEasyHttpRequestDecoder(dispatcher.getDispatcher(), root, protocol));
        channelPipeline.addLast(new RestEasyHttpResponseEncoder());
        if (idleTimeout > 0) {
            channelPipeline.addLast("idleStateHandler", new IdleStateHandler(0, 0, idleTimeout));
        }
        channelPipeline.addLast(eventExecutor, new RequestHandler(dispatcher));
    }

我们可以看到setupHandlers函数中一段关键代码

channelPipeline.addLast(httpChannelHandlers.toArray(new ChannelHandler[httpChannelHandlers.size()]));

这里把一个handlerlist 全部进行了绑定，我们跟踪这个httpChannelHandlers发现它是个空数组，全局搜索引用找到如下方法：

public void setHttpChannelHandlers(final List<ChannelHandler> httpChannelHandlers) {
        this.httpChannelHandlers = httpChannelHandlers == null ? Collections.<ChannelHandler>emptyList() : httpChannelHandlers;
}

看到希望了，这不就是给扩展使用的么，于是开始实现自己StaticServerHandler 对静态文件进行解析

@ChannelHandler.Sharable
public class StaticServerHandler extends SimpleChannelInboundHandler {

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, Object msg) throws Exception {
        if(msg instanceof FullHttpRequest ) {
            FullHttpRequest request= (FullHttpRequest)msg;
            //处理错误或者无法解析的http请求
            String uri = request.uri();
            request.retain();
            routerPaser(ctx, msg, request, uri);
        }
    }

    private void routerPaser(ChannelHandlerContext ctx, Object msg, FullHttpRequest request, String uri) throws IOException {
        if(uri.startsWith("/api/")) {
            //解决前端代理问题
            request.setUri(uri.replaceAll("/api", ""));
            ctx.fireChannelRead(request);
        }else if (uri.startsWith("/h5sign")||uri.startsWith("/mssg2")||uri.startsWith("/mssgApi")) {
            //其他请求往下进行。
            ctx.fireChannelRead(msg);
        }else {
            //页面解析逻辑
            doHander(ctx, request, getPath(ctx, uri));
        }
    }

    private String getPath(ChannelHandlerContext ctx, String uri) {
        String basePath;
        if(null== HomePathUtils.getH5HtmlUrl()||HomePathUtils.getH5HtmlUrl().length()==0) {
            String path = this.getClass().getClassLoader().getResource("templates").getPath();
            if(null==path||path.length()==0){
                notFound404Hander(ctx);
            }
            basePath = path+uri;
        }else{
            //如有设置了H5页面地址，则采用外部页面资源
            basePath = HomePathUtils.getH5HtmlUrl()+uri;
        }
        if(uri.equals(HomePathUtils.getH5StartPath())|| uri.equals(HomePathUtils.getH5JHStartPath())){
            basePath+="index.html";
            return basePath;
        }else if(basePath.contains("?")){
            basePath=basePath.substring(0,basePath.indexOf("?"));
            return basePath;
        }
        return basePath;
    }

    private void doHander(ChannelHandlerContext ctx, FullHttpRequest request, String path) throws IOException {
        File file = new File(path);
        //文件没有发现设置404
        if (!file.exists()) {
            notFound404Hander(ctx);
        }else {
            ctx.write(getHttpResponse(request, path, file));
            RandomAccessFile r = new RandomAccessFile(file, "r");
            ctx.write(new DefaultFileRegion(r.getChannel(), 0, file.length()));
            ChannelFuture channelFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
            if (!HttpUtil.isKeepAlive(request)) {
                channelFuture.addListener(ChannelFutureListener.CLOSE);
            }
            r.close();
        }
    }

    private HttpResponse getHttpResponse(FullHttpRequest request, String path, File file) {
        HttpResponse response = new DefaultHttpResponse(request.protocolVersion(), HttpResponseStatus.OK);
        //设置文件格式内容
        if (path.endsWith(".html")) {
            response.headers().set("Content-Type", "text/html;charset=UTF-8");
        } else if (path.endsWith(".js")) {
            response.headers().set("Content-Type", "application/javascript;charset=UTF-8");
        } else if (path.endsWith(".css")) {
            response.headers().set("Content-Type", "text/css;charset=UTF-8");
        } else if (path.endsWith(".svg")){
            response.headers().set("Content-Type","image/svg+xml");
        } else if (path.endsWith(".png")){
            response.headers().set("Content-Type","image/png");
        } else if (path.endsWith(".ico")){
            response.headers().set("Content-Type","image/x-icon");
        }
        response.headers().set("Accept-Ranges","bytes");
        response.headers().set("Content-Length", file.length());
        response.headers().set("Connection", "keep-alive");
        response.headers().set("Last-Modified",new Date(file.lastModified()));
        try {
            response.headers().set("ETag",generateETagHeaderValue(FileUtils.readFileToByteArray(file),file.lastModified()));
        } catch (IOException e) {
            e.printStackTrace();
        }
        return response;
    }

    public void notFound404Hander(ChannelHandlerContext ctx) {
        FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.NOT_FOUND);
        response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain;charset=UTF-8");
        response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);
        ctx.write(response);
        ChannelFuture channelFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
        channelFuture.addListener(ChannelFutureListener.CLOSE);
    }

    protected StringBuilder generateETagHeaderValue(byte[] bytes,Long str) {
        StringBuilder builder = new StringBuilder(str+"-");
        DigestUtils.appendMd5DigestAsHex(bytes, builder);
        return builder;
    }
}

然后将自己的handler StaticServerHandler交给resteasy 

ArrayList<ChannelHandler> handlers = new ArrayList<>();handlers.add(new StaticServerHandler());netty.setHttpChannelHandlers(handlers);

页面被成功的加载了起来，开始提测，开心的下去叫了杯凉饮。但是没几个小时就被提了几个bug，接着噩梦就开始了。

修成正果

谷歌测试一切正常，但是我们的系统是给政务用的，他们大多都是ie浏览器，netty加载的页面居然在ie上不兼容。

在ie地址栏打开地址什么反应都没有，很奇怪的是偶尔会正常，于是开始f12 一丢丢的看，发现有个js总是加载不完整，从而导致页面展现不出来。

但是偶尔几次可以完整加载，这个文件也不大就1.3MB，开始一通百度，各种方法都尝试了。

Etag也算了以后比之前好多了，成功的频率比之前多了，但是仍然解决根本问题。

开始对比nginx和基于netty实现的静态服务器返回的请求头等是否缺少参数，也都补全了问题依旧。

就这么过了2天自己都想放弃整合页面了，想着干脆用nginx加载算了，就跟领导说实在解决不了了，正在吃早餐的时候突然有个想法从脑中闪过，马上尝试结构ok了。

原来问题出在资源释放哪里，网上例子都是这么释放的，谷歌也正常，但是ie就不行就是如下这段代码

 RandomAccessFile r = new RandomAccessFile(file, "r");
 ctx.write(new DefaultFileRegion(r.getChannel(), 0, file.length()));
 ChannelFuture channelFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
 if (!HttpUtil.isKeepAlive(request)) {
       channelFuture.addListener(ChannelFutureListener.CLOSE);
 }
 r.close();

此处RandomAccessFile 就不用close。如果次数进行了close ie就会出现各种问题，因为netty已经替我们做了释放操作，将此处改为：

 ctx.write(new DefaultFileRegion(file, 0, file.length()));
            ChannelFuture channelFuture = 
 ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);
            if (!HttpUtil.isKeepAlive(request)) {
                channelFuture.addListener(ChannelFutureListener.CLOSE);
 }

其实DefaultFileRegion底层也是直接new的RandomAccessFile

    public void open() throws IOException {
        if (!isOpen() && refCnt() > 0) {
            // Only open if this DefaultFileRegion was not released yet.
            file = new RandomAccessFile(f, "r").getChannel();
        }
    }  

    @Override
    protected void deallocate() {
        FileChannel file = this.file;

        if (file == null) {
            return;
        }
        this.file = null;

        try {
            file.close();
        } catch (IOException e) {
            if (logger.isWarnEnabled()) {
                logger.warn("Failed to close a file.", e);
            }
        }
    }

总结

看来百度的代码不能全信啊，有时候还会给我们带来不必要的麻烦。就拿我这例子来说百度出来的帖子全是一个模板出来的。

所以大家以后还是要多多看优秀框架的源码，了解其底层实现原理，才可以玩的转啊。

---

欢迎关注个人公众号！