TCP 的拆包与粘包
1. 拆包粘包简介
Netty 在基于 TCP 协议的网络通信中,存在拆包与粘包情况。拆包与粘包同时发生在数据的发送方与接收方两方。
发送方通过网络每发送一批二进制数据包,那么这次所发送的数据包就称为一帧,即Frame
。在进行基于 TCP的网络传输时,TCP 协议会将用户真正要发送的数据根据当前缓存的实际情况对其进行拆分或重组,变为用于网络传输的 Frame。在 Netty 中就是将 ByteBuf中的数据拆分或重组为二进制的 Frame。而接收方则需要将接收到的 Frame 中的数据进行重组或拆分,重新恢复为发送方发送时的 ByteBuf 数据。
具体场景描述:
- 发送方发送的 ByteBuf 较大,在传输之前会被 TCP 底层拆分为多个 Frame 进行发送,这个过程称为发送拆包;接收方在接收到需要将这些 Frame 进行合并,这个合并的过程称为接收方粘包。
- 发送方发送的 ByteBuf 较小,无法形成一个 Frame,此时 TCP 底层会将很多的这样的小的 ByteBuf 合并为一个 Frame 进行传输,这个合并的过程称为发送方的粘包;接收方在接收到这个 Frame 后需要进行拆包,拆分出多个原来的小的 ByteBuf,这个拆分的过程称为接收方拆包。
- 当一个 Frame 无法放入整数倍个 ByteBuf 时,最后一个 ByteBuf 就会发生拆包。这个ByteBuf 中的一部分入入到了一个 Frame 中,另一部分被放入到了另一个 Frame 中。这个过程就是发送方拆包。但对于将这些 ByteBuf 放入到一个 Frame 的过程,就是发送方粘包;当接收方在接收到两个 Frame 后,对于第一个 Frame 的最后部分,与第二个 Frame的最前部分会进行合并,这个合并的过程就是接收方粘包。但在将 Frame 中的各个ByteBuf 拆分出来的过程,就是接收方拆包。
纯文字不好理解可以看这个:TCP 粘包/拆包说明 及 异常案例
2. 发送方拆包演示
2.1 需求
客户端作为发送方,向服务端发送两个大的 ByteBuf 数据包
,这两个数据包会被拆分为若干个 Frame 进行发送。这个过程中会发生拆包与粘包。
服务端作为接收方,直接将接收到的 Frame 解码为 String 后进行显示,不对这些 Frame进行粘包与拆包。
2.2 创建工程 03-unpacking
复制 02-socket 工程,在其基础上进行修改,改为:03-unpacking
2.3 定义客户端
(1) 定义客户端启动类
由于客户端仅发送数据,不接收服务端数据,所以这里仅需添加 String 的编码器,用于将字符串编码为 ByteBuf,以在 TCP 上进行传输。
public class SomeClient {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(group)
.channel(NioSocketChannel.class)
.handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new StringEncoder());
pipeline.addLast(new SomeClientHandler());
}
});
ChannelFuture future = bootstrap.connect("localhost", 8888).sync();
future.channel().closeFuture().sync();
} finally {
group.shutdownGracefully();
}
}
}
(2) 定义客户端处理器
在 channelActive()中将数据发送了两次。由于客户端不用读取服务器的数据,所以不用重写 channelRead()方法。
public class SomeClientHandler extends ChannelInboundHandlerAdapter {
private String message = "..";
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
ctx.channel().writeAndFlush(message);
ctx.channel().writeAndFlush(message);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
ctx.close();
}
}
为了能够看到拆包效果,message 的值需要设置的很长,取下面的值。
private String message = "Netty is a NIO client server framework " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"streamlines network programming such as TCP and UDP socket server." +
"'Quick and easy' doesn't mean that a resulting application will " +
"suffer from a maintainability or a performance issue. Netty has " +
"this guide and play with Netty.In other words, Netty is an NIO " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"streamlines network programming such as TCP and UDP socket server." +
"'Quick and easy' doesn't mean that a resulting application will " +
"suffer from a maintainability or a performance issue. Netty has " +
"this guide and play with Netty.In other words, Netty is an NIO " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"streamlines network programming such as TCP and UDP socket server." +
"'Quick and easy' doesn't mean that a resulting application will " +
"suffer from a maintainability or a performance issue. Netty has " +
"this guide and play with Netty.In other words, Netty is an NIO " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"which enables quick and easy development of network applications " +
"such as protocol servers and clients. It greatly simplifies and " +
"streamlines network programming such as TCP and UDP socket server." +
"'Quick and easy' doesn't mean that a resulting application will " +
"suffer from a maintainability or a performance issue. Netty has " +
"this guide and play with Netty.In other words, Netty is an NIO " +
"framework that enables quick and easy development of network " +
"as protocol servers and clients. It greatly simplifies and network " +
"programming such as TCP and UDP socket server development.'Quick " +
"not mean that a resulting application will suffer from a maintain" +
"performance issue. Netty has been designed carefully with the expe " +
"from the implementation of a lot of protocols such as FTP, SMTP, " +
" binary and text-based legacy protocols. As a result, Netty has " +
"a way to achieve of development, performance, stability, without " +
"a compromise.=====================================================";
2.4 定义服务端
(1) 定义服务端启动类
由于服务端仅接收客户端的数据,不发送数据,所以这里仅添加 String 解码器,用于将ByteBuf 解码为 String。
// 定义服务端启动类
public class SomeServer {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup parentGroup = new NioEventLoopGroup();
NioEventLoopGroup childGroup = new NioEventLoopGroup();
try {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(parentGroup, childGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new StringDecoder());
pipeline.addLast(new SomeServerHandler());
}
});
ChannelFuture future = bootstrap.bind(8888).sync();
System.out.println("服务器已启动");
future.channel().closeFuture().sync();
} finally {
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
}
}
(2) 定义服务端处理器
public class SomeServerHandler extends SimpleChannelInboundHandler<String> {
private int counter;
@Override
protected void channelRead0(ChannelHandlerContext ctx, String msg) throws Exception {
System.out.println("Server端接收到的第【" + ++counter + "】个数据包:" + msg);
}
@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
ctx.close();
}
}
演示结果:
先启动服务端,在启动客户端:
去掉编码器,自己编码演示ByteBuf的操作:
Bootstrap bootstrap = new Bootstrap(); bootstrap.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel ch) throws Exception { ChannelPipeline pipeline = ch.pipeline(); // 去掉编码器 // pipeline.addLast(new StringEncoder()); pipeline.addLast(new SomeClientHandler()); } });
@Override public void channelActive(ChannelHandlerContext ctx) throws Exception { byte[] bytes = message.getBytes(); ByteBuf buffer = null; for(int i=0; i<2; i++) { // 申请缓存空间 buffer = Unpooled.buffer(bytes.length); // 将数据写入到缓存 buffer.writeBytes(bytes); // 将缓存中的数据写入到Channel ctx.writeAndFlush(buffer); } }
演示:
结果是一样的
3. 发送方粘包演示
3.1 需求
客户端作为发送方,向服务端发送 100 个小的 ByteBuf
数据包,这 100 个数据包会被合并为若干个 Frame 进行发送。这个过程中会发生粘包与拆包。
服务端作为接收方,直接将接收到的 Frame 解码为 String 后进行显示,不对这些 Frame进行粘包与拆包。
3.2 创建工程 03-stickybag
复制 03-unpacking 工程,在其基础上进行修改:03-stickybag
3.3 定义客户端
(1) 定义客户端启动类
启动类没有变化。
public class SomeClient {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(group)
.channel(NioSocketChannel.class)
.handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new StringEncoder());
pipeline.addLast(new SomeClientHandler());
}
});
ChannelFuture future = bootstrap.connect("localhost", 8888).sync();
future.channel().closeFuture().sync();
} finally {
group.shutdownGracefully();
}
}
}
(2) 定义客户端处理器
把message改成很短的字符串
private String message = "Hello World";
channelActive方法改成如下:
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
for(int i=0; i<100; i++) {
ctx.channel().writeAndFlush(message);
}
}
3.4 定义服务端
服务端没有变化。
演示结果:
先启动服务端,在启动客户端,看到服务端只收到2个数据包,粘包了
去掉编码器,演示ByteBuf的操作:
bootstrap.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<SocketChannel>() { @Override protected void initChannel(SocketChannel ch) throws Exception { ChannelPipeline pipeline = ch.pipeline(); // pipeline.addLast(new StringEncoder()); pipeline.addLast(new SomeClientHandler()); } });
@Override public void channelActive(ChannelHandlerContext ctx) throws Exception { byte[] bytes = message.getBytes(); ByteBuf buffer = null; for(int i=0; i<100; i++) { // 申请缓存空间 buffer = Unpooled.buffer(bytes.length); // 将数据写入到缓存 buffer.writeBytes(bytes); // 将缓存中的数据写入到Channel ctx.writeAndFlush(buffer); } }
演示:结果是一样的
PS:Channel管道这个东西是属于上层的东西,属于Netty里面的东西(也可以说是NIO的),数据从Netty的Channel出去就是网络,是按照Frame,一帧一帧传输的,接收方收到以后也放入自己的管道 ,有的时候在管道中数据量很小,不够组成Frame往外发,就会一直攒,攒够了再发(Frame往外发是TCP协议控制的)
4. 接收方的粘包拆包
为了解决接收方接收到的数据的混乱性,接收方也可以对接收到的 Frame 包进行粘包与拆包。Netty 中已经定义好了很多的接收方粘包拆包解决方案,我们可以直接使用。下面就介绍几个最常用的解决方案。
接收方的粘包拆包实际在做的工作是解码工作。这个解码基本思想
是:
- 发送方在发送数据中添加一个分隔标记,并告诉接收方该标记是什么。这样在接收方接收到 Frame 后,其会根据事先约定好的分隔标记,将数据进行拆分或合并,产生相应的 ByteBuf 数据。这个拆分或合并的过程,称为接收方的拆包与粘包。
5. LineBasedFrameDecoder
基于行的帧解码器,即会按照行分隔符
对数据进行拆包粘包,解码出 ByteBuf。
5.1 创建工程 04-LineBasedFrameDecoder
复制 03-unpacking 工程,在其基础上进行修改:04-LineBasedFrameDecoder
5.2 修改客户端处理器
在发送的消息最后添加行分隔符,其他地方不动。
5.3 修改服务端启动类
行的帧解码器要放在最前面
// 定义服务端启动类
public class SomeServer {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup parentGroup = new NioEventLoopGroup();
NioEventLoopGroup childGroup = new NioEventLoopGroup();
try {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(parentGroup, childGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
//添加行的解码器
pipeline.addLast(new LineBasedFrameDecoder(5120)); // 5k
pipeline.addLast(new StringDecoder());
pipeline.addLast(new SomeServerHandler());
}
});
ChannelFuture future = bootstrap.bind(8888).sync();
System.out.println("服务器已启动");
future.channel().closeFuture().sync();
} finally {
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
}
}
解释:
- 构造需要传入一个参数:
/** * Creates a new decoder. * @param maxLength the maximum length of the decoded frame. * A {@link TooLongFrameException} is thrown if * the length of the frame exceeds this value. * 解码帧的最大长度,如果帧的长度超过这个值, * 就会抛出TooLongFrameException */ public LineBasedFrameDecoder(final int maxLength) { this(maxLength, true, false); }
- 为什么行的帧解码器要放在最前面?
继承体系:
LineBasedFrameDecoder -> ByteToMessageDecoder -> ChannelInboundHandlerAdapter所以要先走LineBasedFrameDecoder,把Frame里面二进制数据变成ByteBuf里面数据,再由StringDecoder把ByteBuf里面数据解码为String数据/** * {@link ChannelInboundHandlerAdapter} which decodes bytes in a stream-like > fashion from one {@link ByteBuf} to an * other Message type. * 它以类似流的方式将字节从一个{@link ByteBuf}解码到另一个消息类型。 * 实际上意思是从Frame帧发来的bytes数据(二进制数据)变成ByteBuf, * 变成ByteBuf以后可以通过类似StringDecoder等其他解码器再变成其他类型 * ... */ public abstract class ByteToMessageDecoder extends ChannelInboundHandlerAdapter {...}
演示效果
6. DelimiterBasedFrameDecoder
基于分隔符的帧解码器,即会按照指定分隔符
对数据进行拆包粘包,解码出 ByteBuf。
6.1 创建工程 04-DelimiterBasedFrameDecoder
复制 03-unpacking 工程,在其基础上进行修改:04-DelimiterBasedFrameDecoder
6.2 修改客户端处理器
在 message 的字符串中多个位置添加任意的分隔符,这里添加的是“###—###”。
6.3 修改服务端启动类
// 定义服务端启动类
public class SomeServer {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup parentGroup = new NioEventLoopGroup();
NioEventLoopGroup childGroup = new NioEventLoopGroup();
try {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(parentGroup, childGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
ByteBuf delimiter = Unpooled.copiedBuffer("###---###".getBytes());
//添加分隔符的解码器,注意分隔符内容需要用ByteBuf类型
pipeline.addLast(new DelimiterBasedFrameDecoder(Integer.MAX_VALUE, delimiter));
pipeline.addLast(new StringDecoder());
pipeline.addLast(new SomeServerHandler());
}
});
ChannelFuture future = bootstrap.bind(8888).sync();
System.out.println("服务器已启动");
future.channel().closeFuture().sync();
} finally {
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
}
}
注意:
- DelimiterBasedFrameDecoder
DelimiterBasedFrameDecoder也是继承ByteToMessageDecoder
演示效果
7. FixedLengthFrameDecoder
固定长度帧解码器,即会按照指定的长度
对 Frame 中的数据进行拆粘包。
7.1 创建工程 04-FixedLengthFrameDecoder
复制 03-stickybag 工程,在其基础上进行修改:04-FixedLengthFrameDecoder
7.2 修改服务端启动类
// 定义服务端启动类
public class SomeServer {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup parentGroup = new NioEventLoopGroup();
NioEventLoopGroup childGroup = new NioEventLoopGroup();
try {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(parentGroup, childGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new FixedLengthFrameDecoder(22));
pipeline.addLast(new StringDecoder());
pipeline.addLast(new SomeServerHandler());
}
});
ChannelFuture future = bootstrap.bind(8888).sync();
System.out.println("服务器已启动");
future.channel().closeFuture().sync();
} finally {
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
}
}
注意:
- 刚好发送的数据长度为11,所以设置11
演示效果
看到刚好100个数据包
- 如果设置长度为22,则有50个数据包
8. LengthFieldBasedFrameDecoder
基于长度域的帧解码器,用于对 LengthFieldPrepender 编码器编码后的数据进行解码
的。所以,首先要清楚 LengthFieldPrepender 编码器的编码原理。(不只LengthFieldPrepender还有其他编码器)
解释:
- LengthFieldPrepender
解释:意思是长度域本身占几个字节
例如:
如果在构造的时候开启了“包含了长度域的长度”
8.1 LengthFieldBasedFrameDecoder构造器参数
maxFrameLength:要解码的 Frame 的最大长度
lengthFieldOffset:长度域的偏移量(一般默认0,就是说长度域不一定要放在最前面,也可以往后移)
lengthFieldLength:长度域的长度
lengthAdjustment:要添加到长度域值中的补偿值,长度矫正值。
initialBytesToStrip:从解码帧中要剥去的前面字节
从解码帧中剥离出的第一字节数,即解码出来以后前面多少字节的数据不要了
- LengthFieldBasedFrameDecoder演示
8.2 创建工程 04-LengthFieldBasedFrameDecoder
复制 02-socket 工程,在其基础上进行修改:04-LengthFieldBasedFrameDecoder
8.3 修改客户端启动类
直接在 02-socket 工程上修改即可。因为是一个C/S通讯的Demo,所以客户端和服务端都需要编码和解码,双方规则要统一
public class SomeClient {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap bootstrap = new Bootstrap();
bootstrap.group(group).channel(NioSocketChannel.class).handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
//编码器,将发出的数据进行编码,添加大小为4个字节的长度域
//长度域中的值不包含长度域本身大小
pipeline.addLast(new LengthFieldPrepender(4));
//接受服务端发来的数据的时候解码器
//解码的最大长度1024
//长度域偏移量为0
//长度域长度为4
//长度矫正值为0
//从解码帧中要剥去的前面字节为4个字节,即去掉长度域
pipeline.addLast(new LengthFieldBasedFrameDecoder(1024,0,4,0,4));
pipeline.addLast(new StringDecoder());
pipeline.addLast(new StringEncoder());
pipeline.addLast(new SomeClientHandler());
}
});
ChannelFuture future = bootstrap.connect("localhost", 8888).sync();
future.channel().closeFuture().sync();
} finally {
group.shutdownGracefully();
}
}
}
8.4 修改服务端启动类
// 定义服务端启动类
public class SomeServer {
public static void main(String[] args) throws InterruptedException {
NioEventLoopGroup parentGroup = new NioEventLoopGroup();
NioEventLoopGroup childGroup = new NioEventLoopGroup();
try {
ServerBootstrap bootstrap = new ServerBootstrap();
bootstrap.group(parentGroup, childGroup)
.channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
//同理不解释了
pipeline.addLast(new LengthFieldBasedFrameDecoder(1024, 0, 4, 0, 4));
pipeline.addLast(new LengthFieldPrepender(4));
pipeline.addLast(new StringDecoder());
pipeline.addLast(new StringEncoder());
pipeline.addLast(new SomeServerHandler());
}
});
ChannelFuture future = bootstrap.bind(8888).sync();
System.out.println("服务器已启动");
future.channel().closeFuture().sync();
} finally {
parentGroup.shutdownGracefully();
childGroup.shutdownGracefully();
}
}
}
结果不演示了,一样的。