正在学习 netty,今天利用所学知识搭建了一个简单的 HTTP 多线程下载服务器。
贴下代码:
先是 ChannelInitializer 的,我用了 netty 自带的处理器搞定 http 协议编解码、聚合还有大块拆分
override fun initChannel(ch: SocketChannel) {
ch.pipeline().addLast("HttpCodec", HttpServerCodec())
ch.pipeline().addLast("HttpAggregator", HttpObjectAggregator(65536))
ch.pipeline().addLast("HttpChunked", ChunkedWriteHandler())
ch.pipeline().addLast("RequestHandle", RequestHandler(basePath))
}
最重要是自己实现的RequestHandler,部分代码如下:
override fun channelRead0(ctx: ChannelHandlerContext, msg: HttpRequest) {
val path = basePath + msg.uri()
logger.debug("Fetching $path")
val file = File(path)
val rfile = RandomAccessFile(file, "r")
if (msg.headers().contains("Range")) { // Multiple threads download
logger.debug("Find 'Range' in HttpRequest: ${msg.headers().get("Range")}")
val range = msg.headers().get("Range").substring(6) // remove 'bytes='
// Line
val response = DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.PARTIAL_CONTENT)
val bIdx = range.substring(0, range.indexOf("-")).toLong()
var eIdx = 0L
if (range.indexOf("-") == range.length - 1) { // example: 0-
// TODO: Judge bIdx > file.size?
eIdx = rfile.length() - 1
} else { // example: 0-100
eIdx = range.substring(range.indexOf("-") + 1, range.length).toLong()
}
logger.debug("bIdx = ${bIdx}; eIdx = ${eIdx}")
// Headers
response.headers().set("Accept-Ranges", "bytes")
response.headers().set("Content-Range", "bytes ${bIdx}-${eIdx}/${rfile.length()}")
response.headers().set("Content-Disposition", "attachment; filename=\"${file.name}\"")
response.headers().set("Content-Type", "application/octet-stream")
response.headers().set("Content-Length", "${eIdx - bIdx + 1}")
// Content
response.content().writeBytes(rfile.channel, bIdx, (eIdx - bIdx + 1).toInt())
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE)
} else { // Single thread download
logger.debug("No 'Range' in HttpRequest")
// Line
val response = DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK)
// Headers
response.headers().set("Accept-Ranges", "bytes")
response.headers().set("Content-Range", "bytes ${0}-${rfile.length() - 1}/${rfile.length()}")
response.headers().set("Content-Disposition", "attachment; filename=\"${file.name}\"")
response.headers().set("Content-Type", "application/octet-stream")
response.headers().set("Content-Length", "${rfile.length()}")
// Content
response.content().writeBytes(rfile.channel, 0L, rfile.length().toInt())
ctx.writeAndFlush(response).addListener(ChannelFutureListener.CLOSE)
}
}
可以看到,自己主要是判断传给我的 HttpRequest 有无 Range,并且设置Content-Length和Accept-Ranges来指明我可以断点续传。
一切运行正常,除了 OOM,运行过程中错误如下:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
at sun.nio.ch.IOUtil.read(IOUtil.java:195)
at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:735)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:718)
at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:292)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1143)
at dd.oliver.htp.RequestHandler.channelRead0(RequestHandler.kt:65)
at dd.oliver.htp.RequestHandler.channelRead0(RequestHandler.kt:16)
这个报错指向了
response.content().writeBytes(rfile.channel, bIdx, (eIdx - bIdx + 1).toInt())
这一行。
我导出 heapdump 并用 mat 查看后确认是某几个 EventLoop 持有高内存非堆内的 buffer 。这种情况按我一开始理解是正常的,因为客户端开太多线程我的服务器也就开了很多 Buffer 。可是人家客户端都没有丝毫问题而我服务器却这样。(内存占用也不一样,我服务器内存占用大概有 5 、6G,而客户端( IDM )只有 500 、600M )
这下我犯难了,我完全没有相关经验不知道从何下手。我想知道有没有大佬有处理经验,如果是你们遇到这样会如何去找问题解决问题呢?