2. 协议设计与解析
2.1 为什么需要协议?
TCP/IP 中消息传输基于流的方式,没有边界。
协议的目的就是划定消息的边界,制定通信双方要共同遵守的通信规则
例如:在网络上传输
下雨天留客天留我不留
是中文一句著名的无标点符号句子,在没有标点符号情况下,这句话有数种拆解方式,而意思却是完全不同,所以常被用作讲述标点符号的重要性
一种解读
下雨天留客,天留,我不留
另一种解读
下雨天,留客天,留我不?留
如何设计协议呢?其实就是给网络传输的信息加上“标点符号”。但通过分隔符来断句不是很好,因为分隔符本身如果用于传输,那么必须加以区分。因此,下面一种协议较为常用
定长字节表示内容长度 + 实际内容
例如,假设一个中文字符长度为 3,按照上述协议的规则,发送信息方式如下,就不会被接收方弄错意思了
0f下雨天留客06天留09我不留
小故事
很久很久以前,一位私塾先生到一家任教。双方签订了一纸协议:“无鸡鸭亦可无鱼肉亦可白菜豆腐不可少不得束修金”。此后,私塾先生虽然认真教课,但主人家则总是给私塾先生以白菜豆腐为菜,丝毫未见鸡鸭鱼肉的款待。私塾先生先是很不解,可是后来也就想通了:主人把鸡鸭鱼肉的钱都会换为束修金的,也罢。至此双方相安无事。
年关将至,一个学年段亦告结束。私塾先生临行时,也不见主人家为他交付束修金,遂与主家理论。然主家亦振振有词:“有协议为证——无鸡鸭亦可,无鱼肉亦可,白菜豆腐不可少,不得束修金。这白纸黑字明摆着的,你有什么要说的呢?”
私塾先生据理力争:“协议是这样的——无鸡,鸭亦可;无鱼,肉亦可;白菜豆腐不可,少不得束修金。”
双方唇枪舌战,你来我往,真个是不亦乐乎!
这里的束修金,也作“束脩”,应当是泛指教师应当得到的报酬
2.2 redis 协议举例
@Slf4j
public class TestRedis {
/* redis协议格式
set name zhangsan
*3
$3
set
$4
name
$8
zhangsan
*/
public static void main(String[] args) {
final byte[] LINE = {13, 10}; //回车换行 \r\n
NioEventLoopGroup worker = new NioEventLoopGroup();
try {
Bootstrap bootstrap = new Bootstrap();
bootstrap.channel(NioSocketChannel.class);
bootstrap.group(worker);
bootstrap.handler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) {
ch.pipeline().addLast(new LoggingHandler());
ch.pipeline().addLast(new ChannelInboundHandlerAdapter() {
@Override
public void channelActive(ChannelHandlerContext ctx) {
ByteBuf buf = ctx.alloc().buffer();
buf.writeBytes("*3".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("$3".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("set".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("$4".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("name".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("$8".getBytes());
buf.writeBytes(LINE);
buf.writeBytes("zhangsan".getBytes());
buf.writeBytes(LINE);
ctx.writeAndFlush(buf);
}
@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
ByteBuf buf = (ByteBuf) msg;
System.out.println(buf.toString(Charset.defaultCharset()));
}
});
}
});
ChannelFuture channelFuture = bootstrap.connect("localhost", 6379).sync();
channelFuture.channel().closeFuture().sync();
} catch (InterruptedException e) {
log.error("client error", e);
} finally {
worker.shutdownGracefully();
}
}
}
2.3 http 协议举例
@Slf4j
public class TestHttp {
public static void main(String[] args) {
NioEventLoopGroup boss = new NioEventLoopGroup();
NioEventLoopGroup worker = new NioEventLoopGroup();
try {
ServerBootstrap serverBootstrap = new ServerBootstrap();
serverBootstrap.channel(NioServerSocketChannel.class);
serverBootstrap.group(boss, worker);
serverBootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
ch.pipeline().addLast(new LoggingHandler(LogLevel.DEBUG));
//http编码器,解码器
ch.pipeline().addLast(new HttpServerCodec());
ch.pipeline().addLast(new SimpleChannelInboundHandler<HttpRequest>() {
@Override
protected void channelRead0(ChannelHandlerContext ctx, HttpRequest msg) throws Exception {
// 获取请求
log.debug(msg.uri());
//返回响应
DefaultFullHttpResponse response = new DefaultFullHttpResponse(msg.protocolVersion(), HttpResponseStatus.OK);
byte[] bytes = "<h1>Hello, world!</h1>".getBytes();
//不加这个,浏览器一直转圈
response.headers().set(CONTENT_LENGTH, bytes.length);
response.content().writeBytes(bytes);
ctx.channel().writeAndFlush(response);
}
});
}
});
ChannelFuture channelFuture = serverBootstrap.bind(8080).sync();
channelFuture.channel().closeFuture().sync();
} catch (Exception e) {
log.error("server error", e);
} finally {
boss.shutdownGracefully();
worker.shutdownGracefully();
}
}
}
浏览器输入:http://127.0.0.1:8080/index,结果
17:26:54 [DEBUG] [nioEventLoopGroup-3-1] i.n.h.l.LoggingHandler - [id: 0x986cb0df, L:/127.0.0.1:8080 - R:/127.0.0.1:55551] REGISTERED
17:26:54 [DEBUG] [nioEventLoopGroup-3-2] i.n.h.l.LoggingHandler - [id: 0x0bc48b95, L:/127.0.0.1:8080 - R:/127.0.0.1:55552] REGISTERED
17:26:54 [DEBUG] [nioEventLoopGroup-3-2] i.n.h.l.LoggingHandler - [id: 0x0bc48b95, L:/127.0.0.1:8080 - R:/127.0.0.1:55552] ACTIVE
17:26:54 [DEBUG] [nioEventLoopGroup-3-1] i.n.h.l.LoggingHandler - [id: 0x986cb0df, L:/127.0.0.1:8080 - R:/127.0.0.1:55551] ACTIVE
17:26:54 [DEBUG] [nioEventLoopGroup-3-1] i.n.h.l.LoggingHandler - [id: 0x986cb0df, L:/127.0.0.1:8080 - R:/127.0.0.1:55551] READ: 662B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 47 45 54 20 2f 69 6e 64 65 78 20 48 54 54 50 2f |GET /index HTTP/|
|00000010| 31 2e 31 0d 0a 48 6f 73 74 3a 20 31 32 37 2e 30 |1.1..Host: 127.0|
|00000020| 2e 30 2e 31 3a 38 30 38 30 0d 0a 43 6f 6e 6e 65 |.0.1:8080..Conne|
|00000030| 63 74 69 6f 6e 3a 20 6b 65 65 70 2d 61 6c 69 76 |ction: keep-aliv|
|00000040| 65 0d 0a 73 65 63 2d 63 68 2d 75 61 3a 20 22 47 |e..sec-ch-ua: "G|
|00000050| 6f 6f 67 6c 65 20 43 68 72 6f 6d 65 22 3b 76 3d |oogle Chrome";v=|
|00000060| 22 31 31 37 22 2c 20 22 4e 6f 74 3b 41 3d 42 72 |"117", "Not;A=Br|
|00000070| 61 6e 64 22 3b 76 3d 22 38 22 2c 20 22 43 68 72 |and";v="8", "Chr|
|00000080| 6f 6d 69 75 6d 22 3b 76 3d 22 31 31 37 22 0d 0a |omium";v="117"..|
|00000090| 73 65 63 2d 63 68 2d 75 61 2d 6d 6f 62 69 6c 65 |sec-ch-ua-mobile|
|000000a0| 3a 20 3f 30 0d 0a 73 65 63 2d 63 68 2d 75 61 2d |: ?0..sec-ch-ua-|
|000000b0| 70 6c 61 74 66 6f 72 6d 3a 20 22 57 69 6e 64 6f |platform: "Windo|
|000000c0| 77 73 22 0d 0a 55 70 67 72 61 64 65 2d 49 6e 73 |ws"..Upgrade-Ins|
|000000d0| 65 63 75 72 65 2d 52 65 71 75 65 73 74 73 3a 20 |ecure-Requests: |
|000000e0| 31 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 4d |1..User-Agent: M|
|000000f0| 6f 7a 69 6c 6c 61 2f 35 2e 30 20 28 57 69 6e 64 |ozilla/5.0 (Wind|
|00000100| 6f 77 73 20 4e 54 20 31 30 2e 30 3b 20 57 69 6e |ows NT 10.0; Win|
|00000110| 36 34 3b 20 78 36 34 29 20 41 70 70 6c 65 57 65 |64; x64) AppleWe|
|00000120| 62 4b 69 74 2f 35 33 37 2e 33 36 20 28 4b 48 54 |bKit/537.36 (KHT|
|00000130| 4d 4c 2c 20 6c 69 6b 65 20 47 65 63 6b 6f 29 20 |ML, like Gecko) |
|00000140| 43 68 72 6f 6d 65 2f 31 31 37 2e 30 2e 30 2e 30 |Chrome/117.0.0.0|
|00000150| 20 53 61 66 61 72 69 2f 35 33 37 2e 33 36 0d 0a | Safari/537.36..|
|00000160| 41 63 63 65 70 74 3a 20 74 65 78 74 2f 68 74 6d |Accept: text/htm|
|00000170| 6c 2c 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 68 |l,application/xh|
|00000180| 74 6d 6c 2b 78 6d 6c 2c 61 70 70 6c 69 63 61 74 |tml+xml,applicat|
|00000190| 69 6f 6e 2f 78 6d 6c 3b 71 3d 30 2e 39 2c 69 6d |ion/xml;q=0.9,im|
|000001a0| 61 67 65 2f 61 76 69 66 2c 69 6d 61 67 65 2f 77 |age/avif,image/w|
|000001b0| 65 62 70 2c 69 6d 61 67 65 2f 61 70 6e 67 2c 2a |ebp,image/apng,*|
|000001c0| 2f 2a 3b 71 3d 30 2e 38 2c 61 70 70 6c 69 63 61 |/*;q=0.8,applica|
|000001d0| 74 69 6f 6e 2f 73 69 67 6e 65 64 2d 65 78 63 68 |tion/signed-exch|
|000001e0| 61 6e 67 65 3b 76 3d 62 33 3b 71 3d 30 2e 37 0d |ange;v=b3;q=0.7.|
|000001f0| 0a 53 65 63 2d 46 65 74 63 68 2d 53 69 74 65 3a |.Sec-Fetch-Site:|
|00000200| 20 6e 6f 6e 65 0d 0a 53 65 63 2d 46 65 74 63 68 | none..Sec-Fetch|
|00000210| 2d 4d 6f 64 65 3a 20 6e 61 76 69 67 61 74 65 0d |-Mode: navigate.|
|00000220| 0a 53 65 63 2d 46 65 74 63 68 2d 55 73 65 72 3a |.Sec-Fetch-User:|
|00000230| 20 3f 31 0d 0a 53 65 63 2d 46 65 74 63 68 2d 44 | ?1..Sec-Fetch-D|
|00000240| 65 73 74 3a 20 64 6f 63 75 6d 65 6e 74 0d 0a 41 |est: document..A|
|00000250| 63 63 65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a 20 |ccept-Encoding: |
|00000260| 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 2c 20 62 |gzip, deflate, b|
|00000270| 72 0d 0a 41 63 63 65 70 74 2d 4c 61 6e 67 75 61 |r..Accept-Langua|
|00000280| 67 65 3a 20 7a 68 2d 43 4e 2c 7a 68 3b 71 3d 30 |ge: zh-CN,zh;q=0|
|00000290| 2e 39 0d 0a 0d 0a |.9.... |
+--------+-------------------------------------------------+----------------+
17:26:54 [DEBUG] [nioEventLoopGroup-3-1] c.i.n.j.c.TestHttp - /index
17:26:54 [DEBUG] [nioEventLoopGroup-3-1] i.n.h.l.LoggingHandler - [id: 0x986cb0df, L:/127.0.0.1:8080 - R:/127.0.0.1:55551] WRITE: 61B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d |HTTP/1.1 200 OK.|
|00000010| 0a 43 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 3a |.Content-length:|
|00000020| 20 32 32 0d 0a 0d 0a 3c 68 31 3e 48 65 6c 6c 6f | 22....<h1>Hello|
|00000030| 2c 20 77 6f 72 6c 64 21 3c 2f 68 31 3e |, world!</h1> |
+--------+-------------------------------------------------+----------------+
2.4 自定义协议要素
- 魔数,用来在第一时间判定是否是无效数据包
- 魔数(Magic Number)通常是一个固定长度的字节序列,用于识别协议或数据结构的开头。魔数的存在可以帮助接收方快速验证数据的有效性,并确保它们理解并按照正确的协议进行处理。
- 版本号,可以支持协议的升级
- 序列化算法,消息正文到底采用哪种序列化反序列化方式,可以由此扩展,例如:json、protobuf、hessian、jdk
- 指令类型,是登录、注册、单聊、群聊… 跟业务相关
- 请求序号,为了双工通信,提供异步能力
- 正文长度
- 消息正文
编解码器
根据上面的要素,设计一个登录请求消息和登录响应消息,并使用 Netty 完成收发
@Slf4j
public class MessageCodec extends ByteToMessageCodec<Message> {
// 编码 出站调用
@Override
protected void encode(ChannelHandlerContext ctx, Message msg, ByteBuf out) throws Exception {
// 4 字节的魔数
out.writeBytes(new byte[]{1, 2, 3, 4});
// 1 字节的版本
out.writeByte(1);
// 1 字节的序列化方式 jdk 0 , json 1
out.writeByte(0);
// 1 字节的指令类型
out.writeByte(msg.getMessageType());
// 4 个字节的请求序号
out.writeInt(msg.getSequenceId());
// 1 字节,无意义,对齐填充,为了将协议头补齐至2的n次幂
out.writeByte(0xff);
// 将message序列化为字节数组
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(msg);
byte[] bytes = bos.toByteArray();
// 4 个字节的正文长度
out.writeInt(bytes.length);
// 消息正文
out.writeBytes(bytes);
}
// 解码 进站调用
@Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
int magicNum = in.readInt();
byte version = in.readByte();
byte serializerType = in.readByte();
byte messageType = in.readByte();
int sequenceId = in.readInt();
in.readByte();
int length = in.readInt();
byte[] bytes = new byte[length];
in.readBytes(bytes, 0, length);
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bytes));
Message message = (Message) ois.readObject();
log.debug("{}, {}, {}, {}, {}, {}", magicNum, version, serializerType, messageType, sequenceId, length);
log.debug("{}", message);
out.add(message);
}
}
测试
public class TestMessageCodec {
public static void main(String[] args) throws Exception {
EmbeddedChannel channel = new EmbeddedChannel(
new LoggingHandler(LogLevel.DEBUG),
new LengthFieldBasedFrameDecoder(
1024, 12, 4, 0, 0
),//解决黏包半包
new MessageCodec()
);
//出站 encode
LoginRequestMessage message = new LoginRequestMessage("zhangsan", "123", "张三");
channel.writeOutbound(message);
//入站 decode
ByteBuf buf = ByteBufAllocator.DEFAULT.buffer();
new MessageCodec().encode(null, message, buf);
//channel.writeInbound(buf);
//模拟半包问题
ByteBuf s1 = buf.slice(0, 100);
ByteBuf s2 = buf.slice(100, buf.readableBytes() - 100);
s1.retain();
channel.writeInbound(s1); // release 1
channel.writeInbound(s2);
}
}
11:02:40 [DEBUG] [main] i.n.h.l.LoggingHandler - [id: 0xembedded, L:embedded - R:embedded] READ: 100B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 01 02 03 04 01 00 00 00 00 00 00 ff 00 00 00 fd |................|
|00000010| ac ed 00 05 73 72 00 36 63 6e 2e 69 74 63 61 73 |....sr.6cn.itcas|
|00000020| 74 2e 6e 65 74 74 79 2e 6a 69 6e 6a 69 65 2e 63 |t.netty.jinjie.c|
|00000030| 32 2e 70 72 6f 74 6f 63 6f 6c 2e 4c 6f 67 69 6e |2.protocol.Login|
|00000040| 52 65 71 75 65 73 74 4d 65 73 73 61 67 65 43 a4 |RequestMessageC.|
|00000050| f9 f0 14 8a f7 ce 02 00 03 4c 00 04 6e 61 6d 65 |.........L..name|
|00000060| 74 00 12 4c |t..L |
+--------+-------------------------------------------------+----------------+
11:02:40 [DEBUG] [main] i.n.h.l.LoggingHandler - [id: 0xembedded, L:embedded - R:embedded] READ COMPLETE
11:02:40 [DEBUG] [main] i.n.h.l.LoggingHandler - [id: 0xembedded, L:embedded - R:embedded] READ: 169B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 6a 61 76 61 2f 6c 61 6e 67 2f 53 74 72 69 6e 67 |java/lang/String|
|00000010| 3b 4c 00 08 70 61 73 73 77 6f 72 64 71 00 7e 00 |;L..passwordq.~.|
|00000020| 01 4c 00 08 75 73 65 72 6e 61 6d 65 71 00 7e 00 |.L..usernameq.~.|
|00000030| 01 78 72 00 2a 63 6e 2e 69 74 63 61 73 74 2e 6e |.xr.*cn.itcast.n|
|00000040| 65 74 74 79 2e 6a 69 6e 6a 69 65 2e 63 32 2e 70 |etty.jinjie.c2.p|
|00000050| 72 6f 74 6f 63 6f 6c 2e 4d 65 73 73 61 67 65 78 |rotocol.Messagex|
|00000060| d7 24 ab 71 27 ac 87 02 00 02 49 00 0b 6d 65 73 |.$.q'.....I..mes|
|00000070| 73 61 67 65 54 79 70 65 49 00 0a 73 65 71 75 65 |sageTypeI..seque|
|00000080| 6e 63 65 49 64 78 70 00 00 00 00 00 00 00 00 74 |nceIdxp........t|
|00000090| 00 06 e5 bc a0 e4 b8 89 74 00 03 31 32 33 74 00 |........t..123t.|
|000000a0| 08 7a 68 61 6e 67 73 61 6e |.zhangsan |
+--------+-------------------------------------------------+----------------+
11:02:40 [DEBUG] [main] c.i.n.j.c.p.MessageCodec - 16909060, 1, 0, 0, 0, 253
11:02:40 [DEBUG] [main] c.i.n.j.c.p.MessageCodec - LoginRequestMessage(super=Message(sequenceId=0, messageType=0), username=zhangsan, password=123, name=张三)
11:02:40 [DEBUG] [main] i.n.h.l.LoggingHandler - [id: 0xembedded, L:embedded - R:embedded] READ COMPLETE
解读
💡 什么时候可以加 @Sharable
- 当 handler 不保存状态时,就可以安全地在多线程下被共享,可以加@Sharable
- 但要注意对于编解码器类,不能继承 ByteToMessageCodec 或 CombinedChannelDuplexHandler 父类,他们的构造方法对 @Sharable 有限制
- 如果能确保编解码器不会保存状态,可以继承 MessageToMessageCodec 父类,再加@Sharable注解
@Slf4j
@ChannelHandler.Sharable
/**
* 必须和 LengthFieldBasedFrameDecoder 一起使用,确保接到的 ByteBuf 消息是完整的
*/
public class MessageCodecSharable extends MessageToMessageCodec<ByteBuf, Message> {
@Override
protected void encode(ChannelHandlerContext ctx, Message msg, List<Object> outList) throws Exception {
ByteBuf out = ctx.alloc().buffer();
// 1. 4 字节的魔数
out.writeBytes(new byte[]{1, 2, 3, 4});
// 2. 1 字节的版本,
out.writeByte(1);
// 3. 1 字节的序列化方式 jdk 0 , json 1
out.writeByte(0);
// 4. 1 字节的指令类型
out.writeByte(msg.getMessageType());
// 5. 4 个字节
out.writeInt(msg.getSequenceId());
// 无意义,对齐填充
out.writeByte(0xff);
// 6. 获取内容的字节数组
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(bos);
oos.writeObject(msg);
byte[] bytes = bos.toByteArray();
// 7. 长度
out.writeInt(bytes.length);
// 8. 写入内容
out.writeBytes(bytes);
outList.add(out);
}
@Override
protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
int magicNum = in.readInt();
byte version = in.readByte();
byte serializerType = in.readByte();
byte messageType = in.readByte();
int sequenceId = in.readInt();
in.readByte();
int length = in.readInt();
byte[] bytes = new byte[length];
in.readBytes(bytes, 0, length);
ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(bytes));
Message message = (Message) ois.readObject();
log.debug("{}, {}, {}, {}, {}, {}", magicNum, version, serializerType, messageType, sequenceId, length);
log.debug("{}", message);
out.add(message);
}
}