Antrea-数据流分析

最新推荐文章于 2022-12-06 12:00:43 发布
最新推荐文章于 2022-12-06 12:00:43 发布
阅读量723 收藏
点赞数
文章标签： kubernetes antrea cni
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43394724/article/details/118640174
版权

本实验接上一篇Antrea-安装与拓扑
本篇讨论在该拓扑下数据流情况
本实验地址表

NamePortIP AddrMACNote
worker-01 br-int(OVS)turn0(1)N/A06:c6:e4:f2:94:51
worker-01 br-int(OVS)gw0(2)10.211.1.136🆎6b:31:0c:75
worker-01 br-int(OVS)veth(15)N/A86:e0:33:27:a8:c0to pod frountend
worker-01 br-int(OVS)veth(16)N/Ac6:97:94:e1:2e:a5to pod backend1
pod-frontendeth0(15)10.211.1.142a:42:47:c6:9b:15
pod-backend1eth0(16)10.211.1.1586:5c:0a:d2:c8:c3
worker-01ens192192.168.110.6600:50:56:b2:35:31
worker-02 br-int(OVS)turn0(1)N/A0a:80:b4:dc:33:4c
worker-02 br-int(OVS)gw0(2)10.211.2.112:c8:e9:86:7e:7b
worker-02 br-int(OVS)veth(14)N/Ae6:bf:f2:24:d4:f2to pod backend2
pod-backend2eth0(14)10.211.2.3012:c2:26:2b:46:77
worker-02ens192192.168.110.6700:50:56:b2:46:58

OVS Pipeline

OVS Pipeline是数据进入OVS后的判别及流程，通过一系列Table完成
在这里插入图片描述
在实验中，我们选取一段流进行分析：

Frontend to Service backendsvc

拓扑

在这里插入图片描述

[root@master-01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
backendsvc   ClusterIP   10.101.216.214   <none>        80/TCP    4h45m
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   4d21h

我们知道，service backendsvc的地址为：10.101.216.214

进入pod frontend监控包

[root@master-01 ~]# kubectl  exec -it frontend -- sh
/ # tcpdump -en
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

在另外一个终端（@master-01)上运行以下命令以访问服务backendsvc

[root@master-01 ~]# kubectl exec -it frontend -c frontend  -- curl backendsvc
Praqma Network MultiTool (with NGINX) - backend2 - 10.211.2.30
[root@master-01 ~]# kubectl exec -it frontend -c frontend  -- curl backendsvc
Praqma Network MultiTool (with NGINX) - backend1 - 10.211.1.15

观察pod的监控，取svc指向backend1时，得到抓包结果：

12:27:02.979220 2a:42:47:c6:9b:15 > 36:ab:6b:31:0c:75, ethertype IPv4 (0x0800), length 96: 10.211.1.14.38861 > 10.96.0.10.53: 32140+ A? backendsvc.default.svc.cluster.local. (54)
12:27:02.980269 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 96: 10.211.1.14.38861 > 10.96.0.10.53: 32464+ AAAA? backendsvc.default.svc.cluster.local. (54)
12:27:02.981453 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 189: 10.96.0.10.53 > 10.211.1.14.38861: 32464*- 0/1/0 (147)
12:27:02.981470 36:ab:6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 148: 10.96.0.10.53 > 10.211.1.14.38861: 32140*- 1/0/0 A 10.101.216.214 (106)
12:27:02.981770 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 74: 10.211.1.14.48078 > 10.101.216.214.80: Flags [S], seq 2351183631, win 28200, options [mss 1410,sackOK,TS val 251618365 ecr 0,nop,wscale 7], length 0
12:27:02.982979 36:ab:6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 74: 10.101.216.214.80 > 10.211.1.14.48078: Flags [S.], seq 343862012, ack 2351183632, win 27960, options [mss 1410,sackOK,TS val 251618366 ecr 251618365,nop,wscale 7], length 0
12:27:02.983054 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 66: 10.211.1.14.48078 > 10.101.216.214.80: Flags [.], ack 1, win 221, options [nop,nop,TS val 251618367 ecr 251618366], length 0
12:27:02.983158 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 140: 10.211.1.14.48078 > 10.101.216.214.80: Flags [P.], seq 1:75, ack 1, win 221, options [nop,nop,TS val 251618367 ecr 251618366], length 74: HTTP: GET / HTTP/1.1
12:27:02.983201 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 66: 10.101.216.214.80 > 10.211.1.14.48078: Flags [.], ack 75, win 219, options [nop,nop,TS val 251618367 ecr 251618367], length 0
12:27:02.984074 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 302: 10.101.216.214.80 > 10.211.1.14.48078: Flags [P.], seq 1:237, ack 75, win 219, options [nop,nop,TS val 251618368 ecr 251618367], length 236: HTTP: HTTP/1.1 200 OK
12:27:02.984085 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 66: 10.211.1.14.48078 > 10.101.216.214.80: Flags [.], ack 237, win 229, options [nop,nop,TS val 251618368 ecr 251618368], length 0
12:27:02.984117 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 129: 10.101.216.214.80 > 10.211.1.14.48078: Flags [P.], seq 237:300, ack 75, win 219, options [nop,nop,TS val 251618368 ecr 251618368], length 63: HTTP
12:27:02.984122 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 66: 10.211.1.14.48078 > 10.101.216.214.80: Flags [.], ack 300, win 229, options [nop,nop,TS val 251618368 ecr 251618368], length 0
12:27:02.984374 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 66: 10.211.1.14.48078 > 10.101.216.214.80: Flags [F.], seq 75, ack 300, win 229, options [nop,nop,TS val 251618368 ecr 251618368], length 0
12:27:02.984488 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 66: 10.101.216.214.80 > 10.211.1.14.48078: Flags [F.], seq 300, ack 76, win 219, options [nop,nop,TS val 251618368 ecr 251618368], length 0
12:27:02.984500 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 66: 10.211.1.14.48078 > 10.101.216.214.80: Flags [.], ack 301, win 229, options [nop,nop,TS val 251618368 ecr 251618368], length 0
12:27:07.992440 86:5c:0a:d2:c8:c3 > 2a:42:47:c6:9b:15, ethertype ARP (0x0806), length 42: Request who-has 10.211.1.14 tell 10.211.1.15, length 28
12:27:07.992453 2a:42:47:c6:9b:15 > 86:5c:0a:d2:c8:c3, ethertype ARP (0x0806), length 42: Reply 10.211.1.14 is-at 2a:42:47:c6:9b:15, length 28

  • 12:27:02.979220 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 96: 10.211.1.14.38861 > 10.96.0.10.53: 32140+ A? backendsvc.default.svc.cluster.local. (54)
    12:27:02.980269 2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75, ethertype IPv4 (0x0800), length 96: 10.211.1.14.38861 > 10.96.0.10.53: 32464+ AAAA? backendsvc.default.svc.cluster.local. (54)
    Pod/frontend询问coreDNS backendsvc.default.svc.cluster.local的地址
    对Pod来查看DNS Server，就是10.96.0.10
/ # nslookup backendsvc
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   backendsvc.default.svc.cluster.local
Address: 10.101.216.214

流量在OVS里面2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75，即veth(15)–>gw0(2)

  • 12:27:02.981470 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 148: 10.96.0.10.53 > 10.211.1.14.38861: 32140- 1/0/0 A 10.101.216.214 (106)*
    coreDNS回给pod/frontend DNS应答，流量在OVS里面36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15，即gw0(2)–>veth(15)
  • 12:27:02.981470 36🆎6b:31:0c:75 > 2a:42:47:c6:9b:15, ethertype IPv4 (0x0800), length 148: 10.96.0.10.53 > 10.211.1.14.38861: 32140- 1/0/0 A 10.101.216.214 (106)*
    pod/frontend开始向service backendsvc发送访问连接，流量在OVS里面2a:42:47:c6:9b:15 > 36🆎6b:31:0c:75，即veth(15)–>gw0(2)

Table流表分析

Table#0

可以通过以下命令查看worker-01上的Table#0

[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs – ovs-ofctl dump-flows br-int table=0 --no-stats
cookie=0x1000000000000, priority=200,in_port=“antrea-gw0” actions=load:0x1->NXM_NX_REG0[0…15],resubmit(,10)
cookie=0x1000000000000, priority=200,in_port=“antrea-tun0” actions=load:0->NXM_NX_REG0[0…15],load:0x1->NXM_NX_REG0[19],resubmit(,30)
cookie=0x1030000000000, priority=190,in_port=“coredns–c4f3d4” actions=load:0x2->NXM_NX_REG0[0…15],resubmit(,10)
cookie=0x1030000000000, priority=190,in_port=“backend1-911dea” actions=load:0x2->NXM_NX_REG0[0…15],resubmit(,10)
cookie=0x1030000000000, priority=190,in_port="frontend-fbd015" actions=load:0x2->NXM_NX_REG0[0..15],resubmit(,10)
cookie=0x1000000000000, priority=0 actions=drop

其中，对当前流进行分类； “0”代表隧道，“1”代表本地网关，“2”代表本地 pod。如上面例子中我们用到：
cookie=0x1030000000000, priority=190,in_port=“frontend-fbd015” actions=load:0x2->NXM_NX_REG0[0…15],resubmit(,10)
此流条目中的第一个动作是将寄存器 reg0[0…15] 的值设置为“0x2”，这意味着流来自本地 pod。 同一流条目中的第二个动作是将流移交给下一个表，即Table#10。 (resubmit(,10)) ，参考上一节中的Pipeline。

Table#10

[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs – ovs-ofctl dump-flows br-int table=10 --no-stats
cookie=0x1000000000000, table=10, priority=200,ipv6,ipv6_src=fe80::/10 actions=resubmit(,21)
cookie=0x1000000000000, table=10, priority=200,ip,in_port=“antrea-gw0” actions=resubmit(,29)
cookie=0x1000000000000, table=10, priority=200,arp,in_port=“antrea-gw0”,arp_spa=10.211.1.1,arp_sha=36🆎6b:31:0c:75 actions=resubmit(,20)
cookie=0x1030000000000, table=10, priority=200,arp,in_port=“coredns–c4f3d4”,arp_spa=10.211.1.4,arp_sha=92:d8:7a:ad:79:c9 actions=resubmit(,20)
cookie=0x1030000000000, table=10, priority=200,arp,in_port=“backend1-911dea”,arp_spa=10.211.1.15,arp_sha=86:5c:0a:d2:c8:c3 actions=resubmit(,20)
cookie=0x1030000000000, table=10, priority=200,arp,in_port="frontend-fbd015",arp_spa=10.211.1.14,arp_sha=2a:42:47:c6:9b:15 actions=resubmit(,20)
cookie=0x1030000000000, table=10, priority=200,ip,in_port=“coredns–c4f3d4”,dl_src=92:d8:7a:ad:79:c9,nw_src=10.211.1.4 actions=resubmit(,29)
cookie=0x1030000000000, table=10, priority=200,ip,in_port=“backend1-911dea”,dl_src=86:5c:0a:d2:c8:c3,nw_src=10.211.1.15 actions=resubmit(,29)
cookie=0x1030000000000, table=10, priority=200,ip,in_port="frontend-fbd015",dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14 actions=resubmit(,29)
cookie=0x1000000000000, table=10, priority=0 actions=drop

和frontend相关的两条：

  • cookie=0x1030000000000, table=10, priority=200,arp,in_port=“frontend-fbd015”,arp_spa=10.211.1.14,arp_sha=2a:42:47:c6:9b:15 actions=resubmit(,20)
    arp的包，地址为10.211.1.14，转到table#20

  • cookie=0x1030000000000, table=10, priority=200,ip,in_port=“frontend-fbd015”,dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14 actions=resubmit(,29)
    IP包且源地址为10.211.1.14，转到table#29

Table#30

table#29–>table#30，直接看table#30的情况

[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs -- ovs-ofctl dump-flows br-int table=30 --no-stats
 cookie=0x1000000000000, table=30, priority=200,ip actions=ct(table=31,zone=65520,nat)
 cookie=0x1000000000000, table=30, priority=200,ipv6 actions=ct(table=31,zone=65510,nat)

Conntrack 表的工作是开始跟踪所有流量。 （“ct”动作表示连接跟踪connection tracking）任何通过此表的流都将处于“已跟踪”（trk）状态。 在通用网络安全术语中，此功能使 OVS 成为有状态的连接感知组件。
流程然后被移交给下一个表，即table#31； 如所示 (actions=ct(table=31,))。 下一站是table#31。

Table#31

[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs – ovs-ofctl dump-flows br-int table=31 --no-stats
cookie=0x1040000000000, table=31, priority=200,ct_state=-new+trk,ct_mark=0x21,ip actions=load:0x1->NXM_NX_REG0[19],resubmit (,50)
cookie=0x1000000000000, table=31, priority=190,ct_state=+inv+trk,ip actions=drop
cookie=0x1040000000000, table=31, priority=190,ct_state=-new+trk,ip actions=resubmit(,50)
cookie=0x1000000000000, table=31, priority=0 actions=resubmit(,40),resubmit(,41)

Table#31是ConntrackState 表，它处理所有处于跟踪状态的流（基本上由 Conntrack 表 table#30 移交）。 上面显示的第一个和第三个流条目处理流不是新的且被跟踪的流。 （ct_state=-new 表示不是新的，+trk 表示被跟踪）第二个流条目处理相应流为 INVALID 和 TRACKED 的流，基本上它丢弃所有这些流。

测试的流是pod/frontend 到backendsvc 服务的全新流程。 因此，当前流将匹配上面流表中的最后一个条目。 该流条目中的操作是将流移交给下一个表 (resubmit(,40)),resubmit(,41) table#40/41。

Table#40/41

[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs – ovs-ofctl dump-flows br-int table=40 --no-stats
cookie=0x1040000000000, table=40, priority=0 actions=load:0x1->NXM_NX_REG4[16…18]
[root@master-01 ~]# kubectl exec -n kube-system -it antrea-agent-bj42h -c antrea-ovs – ovs-ofctl dump-flows br-int table=41 --no-stats
cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.1,tp_dst=443 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:2
cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.119.154,tp_dst=443 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:3
cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.10,tp_dst=53 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:5
cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.96.0.10,tp_dst=9153 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:6
cookie=0x1040000000000, table=41, priority=200,udp,reg4=0x10000/0x70000,nw_dst=10.96.0.10,tp_dst=53 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:4
cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.101.216.214,tp_dst=80 actions=load:0x2->NXM_NX_REG4[16..18],load:0x1->NXM_NX_REG0[19],group:1
cookie=0x1000000000000, table=41, priority=0 actions=resubmit(,42)

cookie=0x1040000000000, table=41, priority=200,tcp,reg4=0x10000/0x70000,nw_dst=10.101.216.214,tp_dst=80 actions=load:0x2->NXM_NX_REG4[16…18],load:0x1->NXM_NX_REG0[19],group:1
该表把转到了gw0，在Pipeline的如下部分：
在这里插入图片描述

IPtables

Worker-01 节点的 Kernel IP stack 接收来自 antrea-gw0 接口的流，流由 kube-proxy 管理的 iptables NAT 规则处理。 Worker -01 节点上的 iptables NAT 规则如下所示。

  • “-t nat” pulls the NAT table
  • “-L” lists all the rules in the given table
  • “-n” displayes IP address and port numbers (rather than hostnames and service names)

[root@worker-01 ~]# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all – 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals /
DOCKER all – 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all – 0.0.0.0/0 0.0.0.0/0 / kubernetes service portals /
DOCKER all – 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all – 0.0.0.0/0 0.0.0.0/0 / kubernetes postrouting rules /
MASQUERADE all – 172.17.0.0/16 0.0.0.0/0
RETURN all – 192.168.122.0/24 224.0.0.0/24
RETURN all – 192.168.122.0/24 255.255.255.255
MASQUERADE tcp – 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp – 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all – 192.168.122.0/24 !192.168.122.0/24
ANTREA-POSTROUTING all – 0.0.0.0/0 0.0.0.0/0 / Antrea: jump to Antrea postrouting rules /
Chain ANTREA-POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all – 10.211.1.0/24 0.0.0.0/0 / Antrea: masquerade pod to external packets / ! match-set ANTREA-POD-IP dst
Chain DOCKER (2 references)
target prot opt source destination
RETURN all – 0.0.0.0/0 0.0.0.0/0
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
Chain KUBE-MARK-DROP (0 references)
target prot opt source destination
MARK all – 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000
Chain KUBE-MARK-MASQ (16 references)
target prot opt source destination
MARK all – 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
Chain KUBE-POSTROUTING (1 references)
target prot opt source destination
RETURN all – 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000
MARK all – 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000
MASQUERADE all – 0.0.0.0/0 0.0.0.0/0 / kubernetes service traffic requiring SNAT /
Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination
Chain KUBE-SEP-73HF7GS7WMEQK5M6 (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.1.4 0.0.0.0/0 / kube-system/kube-dns:dns-tcp /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns-tcp / tcp to:10.211.1.4:53
Chain KUBE-SEP-AB7YO7A3CB2CZWK5 (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.1.4 0.0.0.0/0 / kube-system/kube-dns:dns /
DNAT udp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns / udp to:10.211.1.4:53
Chain KUBE-SEP-BQS327LSMHNPOPC6 (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 10.211.2.30 0.0.0.0/0 /* default/backendsvc */ DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/backendsvc */ tcp to:10.211.2.30:80
Chain KUBE-SEP-DFFSKWBTAX72RCQG (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 192.168.110.66 0.0.0.0/0 / kube-system/antrea /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / kube-system/antrea / tcp to:192.168.110.66:10349
Chain KUBE-SEP-EQY7OT4FTLCGYYAP (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.2.21 0.0.0.0/0 / kube-system/kube-dns:dns-tcp /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns-tcp / tcp to:10.211.2.21:53
Chain KUBE-SEP-GH55AAVWO6AS4CDN (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.2.21 0.0.0.0/0 / kube-system/kube-dns:metrics /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:metrics / tcp to:10.211.2.21:9153
Chain KUBE-SEP-GNMIPSQVBZ6K56T3 (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 10.211.1.15 0.0.0.0/0 /* default/backendsvc */ DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* default/backendsvc */ tcp to:10.211.1.15:80
Chain KUBE-SEP-JBDJ3O64IUA4B6TG (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.2.21 0.0.0.0/0 / kube-system/kube-dns:dns /
DNAT udp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns / udp to:10.211.2.21:53
Chain KUBE-SEP-N3D6XPNX4VV2UWVN (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 192.168.110.61 0.0.0.0/0 / default/kubernetes:https /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / default/kubernetes:https / tcp to:192.168.110.61:6443
Chain KUBE-SEP-VCCKD2O5C6GIOK2X (1 references)
target prot opt source destination
KUBE-MARK-MASQ all – 10.211.1.4 0.0.0.0/0 / kube-system/kube-dns:metrics /
DNAT tcp – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:metrics / tcp to:10.211.1.4:9153
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-MARK-MASQ tcp – !10.211.0.0/16 10.96.0.1 / default/kubernetes:https cluster IP / tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y tcp – 0.0.0.0/0 10.96.0.1 / default/kubernetes:https cluster IP / tcp dpt:443
KUBE-MARK-MASQ tcp – !10.211.0.0/16 10.96.119.154 / kube-system/antrea cluster IP / tcp dpt:443
KUBE-SVC-QDWG4LJGNBTOT5ED tcp – 0.0.0.0/0 10.96.119.154 / kube-system/antrea cluster IP / tcp dpt:443
KUBE-MARK-MASQ udp – !10.211.0.0/16 10.96.0.10 / kube-system/kube-dns:dns cluster IP / udp dpt:53
KUBE-SVC-TCOU7JCQXEZGVUNU udp – 0.0.0.0/0 10.96.0.10 / kube-system/kube-dns:dns cluster IP / udp dpt:53
KUBE-MARK-MASQ tcp – !10.211.0.0/16 10.96.0.10 / kube-system/kube-dns:dns-tcp cluster IP / tcp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4 tcp – 0.0.0.0/0 10.96.0.10 / kube-system/kube-dns:dns-tcp cluster IP / tcp dpt:53
KUBE-MARK-MASQ tcp – !10.211.0.0/16 10.96.0.10 / kube-system/kube-dns:metrics cluster IP / tcp dpt:9153
KUBE-SVC-JD5MR3NA4I4DYORP tcp – 0.0.0.0/0 10.96.0.10 / kube-system/kube-dns:metrics cluster IP / tcp dpt:9153
KUBE-MARK-MASQ tcp -- !10.211.0.0/16 10.101.216.214 /* default/backendsvc cluster IP */ tcp dpt:80 KUBE-SVC-SQRBXEDPJTJWMLH3 tcp -- 0.0.0.0/0 10.101.216.214 /* default/backendsvc cluster IP */ tcp dpt:80
KUBE-NODEPORTS all – 0.0.0.0/0 0.0.0.0/0 / kubernetes service nodeports; NOTE: this must be the last rule in this chain / ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
target prot opt source destination
KUBE-SEP-73HF7GS7WMEQK5M6 all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns-tcp / statistic mode random probability 0.50000000000
KUBE-SEP-EQY7OT4FTLCGYYAP all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns-tcp /
Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
target prot opt source destination
KUBE-SEP-VCCKD2O5C6GIOK2X all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:metrics / statistic mode random probability 0.50000000000
KUBE-SEP-GH55AAVWO6AS4CDN all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:metrics /
Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target prot opt source destination
KUBE-SEP-N3D6XPNX4VV2UWVN all – 0.0.0.0/0 0.0.0.0/0 / default/kubernetes:https /
Chain KUBE-SVC-QDWG4LJGNBTOT5ED (1 references)
target prot opt source destination
KUBE-SEP-DFFSKWBTAX72RCQG all – 0.0.0.0/0 0.0.0.0/0 / kube-system/antrea /
Chain KUBE-SVC-SQRBXEDPJTJWMLH3 (1 references)
target prot opt source destination
KUBE-SEP-GNMIPSQVBZ6K56T3 all – 0.0.0.0/0 0.0.0.0/0 / default/backendsvc / statistic mode random probability 0.50000000000
KUBE-SEP-BQS327LSMHNPOPC6 all – 0.0.0.0/0 0.0.0.0/0 / default/backendsvc /
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
target prot opt source destination
KUBE-SEP-AB7YO7A3CB2CZWK5 all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns / statistic mode random probability 0.50000000000
KUBE-SEP-JBDJ3O64IUA4B6TG all – 0.0.0.0/0 0.0.0.0/0 / kube-system/kube-dns:dns */

摘取相关条目

Chain KUBE-SEP-BQS327LSMHNPOPC6 (1 references) 
target     prot opt source               destination 
KUBE-MARK-MASQ  all  --  10.211.2.30          0.0.0.0/0            /* default/backendsvc */ 
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/backendsvc */ tcp to:10.211.2.30:80

Chain KUBE-SEP-GNMIPSQVBZ6K56T3 (1 references)
target     prot opt source               destination
KUBE-MARK-MASQ  all  --  10.211.1.15          0.0.0.0/0            /* default/backendsvc */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/backendsvc */ tcp to:10.211.1.15:80

KUBE-MARK-MASQ  tcp  -- !10.211.0.0/16        10.101.216.214       /* default/backendsvc cluster IP */ tcp dpt:80
KUBE-SVC-SQRBXEDPJTJWMLH3  tcp  --  0.0.0.0/0            10.101.216.214       /* default/backendsvc cluster IP */ tcp dpt:80

在流向 backendsvc 服务 IP 的流上应用DNAT，并选择 backend1 pod 10.211.1.15 或 backend2 pod 10.211.2.30作为目的地。
这是典型的 kube-proxy 管理、iptables 驱动的“ClusterIP”类型的 Kubernetes 服务功能，它为 Kubernetes 集群内的流提供分布式负载平衡。

在Antrea pod上查看
# antctl trace-packet -S default/frontend -D default/backendsvc -f "tcp,tcp_dst=80"
Fow: tcp,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=36:ab:6b:31:0c:75,nw_src=10.211.1.14,nw_dst=10.101.216.214,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0

bridge("br-int")
----------------
 0. in_port=15, priority 190, cookie 0x1030000000000
    load:0x2->NXM_NX_REG0[0..15]
    goto_table:10
10. ip,in_port=15,dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14, priority 200, cookie 0x1030000000000
    goto_table:29
29. priority 0, cookie 0x1000000000000
    goto_table:30
30. ip, priority 200, cookie 0x1000000000000
    ct(table=31,zone=65520,nat)
    nat
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 31.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: tcp,reg0=0x2,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=36:ab:6b:31:0c:75,nw_src=10.211.1.14,nw_dst=10.101.216.214,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0
Megaflow: recirc_id=0,eth,ip,in_port=15,dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14,nw_dst=0.0.0.0/1,nw_frag=no
Datapath actions: ct(zone=65520,nat),recirc(0x2f1e)

===============================================================================
recirc(0x2f1e) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
Replacing src/dst IP/ports to simulate NAT:
 Initial flow:
 Modified flow:
===============================================================================

Flow: recirc_id=0x2f1e,ct_state=new|trk,ct_zone=65520,eth,tcp,reg0=0x2,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=36:ab:6b:31:0c:75,nw_src=10.211.1.14,nw_dst=10.101.216.214,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0

bridge("br-int")
----------------
    thaw
        Resuming from table 31
31. priority 0, cookie 0x1000000000000
    resubmit(,40)
    40. priority 0, cookie 0x1040000000000
            load:0x1->NXM_NX_REG4[16..18]
    resubmit(,41)
41. tcp,reg4=0x10000/0x70000,nw_dst=10.101.216.214,tp_dst=80, priority 200, cookie 0x1040000000000
    load:0x2->NXM_NX_REG4[16..18]
    load:0x1->NXM_NX_REG0[19]
    group:1
     -> no live bucket

Final flow: recirc_id=0x2f1e,ct_state=new|trk,ct_zone=65520,eth,tcp,reg0=0x80002,reg4=0x20000,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=36:ab:6b:31:0c:75,nw_src=10.211.1.14,nw_dst=10.101.216.214,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=80,tcp_flags=0
Megaflow: recirc_id=0x2f1e,ct_state=+new-inv+trk,ct_mark=0,eth,tcp,in_port=15,nw_dst=10.101.216.214,nw_frag=no,tp_dst=80
Datapath actions: hash(l4(0)),recirc(0x2f1f)

此过程和我们前面分析的一致。

查看frontend到backend1的全过程

# antctl trace-packet -S default/frontend -D default/backend1 -f "tcp,tcp_src=80"
Flow: tcp,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=86:5c:0a:d2:c8:c3,nw_src=10.211.1.14,nw_dst=10.211.1.15,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0

bridge("br-int")
----------------
 0. in_port=15, priority 190, cookie 0x1030000000000
    load:0x2->NXM_NX_REG0[0..15]
    goto_table:10
10. ip,in_port=15,dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14, priority 200, cookie 0x1030000000000
    goto_table:29
29. priority 0, cookie 0x1000000000000
    goto_table:30
30. ip, priority 200, cookie 0x1000000000000
    ct(table=31,zone=65520,nat)
    nat
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 31.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: tcp,reg0=0x2,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=86:5c:0a:d2:c8:c3,nw_src=10.211.1.14,nw_dst=10.211.1.15,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0
Megaflow: recirc_id=0,eth,ip,in_port=15,dl_src=2a:42:47:c6:9b:15,nw_src=10.211.1.14,nw_dst=0.0.0.0/1,nw_frag=no
Datapath actions: ct(zone=65520,nat),recirc(0x2f3e)

===============================================================================
recirc(0x2f3e) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
Replacing src/dst IP/ports to simulate NAT:
 Initial flow:
 Modified flow:
===============================================================================

Flow: recirc_id=0x2f3e,ct_state=new|trk,ct_zone=65520,eth,tcp,reg0=0x2,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=86:5c:0a:d2:c8:c3,nw_src=10.211.1.14,nw_dst=10.211.1.15,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0

bridge("br-int")
----------------
    thaw
        Resuming from table 31
31. priority 0, cookie 0x1000000000000
    resubmit(,40)
    40. priority 0, cookie 0x1040000000000
            load:0x1->NXM_NX_REG4[16..18]
    resubmit(,41)
41. priority 0, cookie 0x1000000000000
    goto_table:42
42. priority 0, cookie 0x1000000000000
    goto_table:50
50. priority 0, cookie 0x1000000000000
    goto_table:60
60. priority 0, cookie 0x1000000000000
    goto_table:61
61. priority 0, cookie 0x1000000000000
    goto_table:70
70. priority 0, cookie 0x1000000000000
    goto_table:80
80. dl_dst=86:5c:0a:d2:c8:c3, priority 200, cookie 0x1030000000000
    load:0x10->NXM_NX_REG1[]
    load:0x1->NXM_NX_REG0[16]
    goto_table:90
90. priority 0, cookie 0x1000000000000
    goto_table:100
100. priority 0, cookie 0x1000000000000
    goto_table:101
101. priority 0, cookie 0x1000000000000
    goto_table:105
105. ct_state=+new+trk,ip, priority 190, cookie 0x1000000000000
    ct(commit,table=106,zone=65520)
    drop
     -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 106.
     -> Sets the packet to an untracked state, and clears all the conntrack fields.

Final flow: recirc_id=0x2f3e,eth,tcp,reg0=0x10002,reg1=0x10,reg4=0x10000,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=86:5c:0a:d2:c8:c3,nw_src=10.211.1.14,nw_dst=10.211.1.15,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0
Megaflow: pkt_mark=0/0x1,recirc_id=0x2f3e,ct_state=+new-est-rpl-inv+trk,ct_mark=0,eth,ip,in_port=15,dl_dst=86:5c:0a:d2:c8:c3,nw_dst=10.211.1.0/24,nw_frag=no
Datapath actions: ct(commit,zone=65520),recirc(0x2f3f)

===============================================================================
recirc(0x2f3f) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================

Flow: recirc_id=0x2f3f,ct_state=new|trk,ct_zone=65520,eth,tcp,reg0=0x10002,reg1=0x10,reg4=0x10000,in_port=15,vlan_tci=0x0000,dl_src=2a:42:47:c6:9b:15,dl_dst=86:5c:0a:d2:c8:c3,nw_src=10.211.1.14,nw_dst=10.211.1.15,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0

bridge("br-int")
----------------
    thaw
        Resuming from table 106
106. priority 0, cookie 0x1000000000000
    goto_table:110
110. ip,reg0=0x10000/0x10000, priority 200, cookie 0x1000000000000
    output:NXM_NX_REG1[]
     -> output port is 16

Final flow: unchanged
Megaflow: recirc_id=0x2f3f,eth,ip,in_port=15,nw_src=10.211.1.14,nw_frag=no
Datapath actions: 4

以上

Etaon
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
理解 OpenStack 高可用（HA）：Neutron 分布式虚拟路由（下）
qq_34043421的博客
05-25 1153
1. 代码分析DVR 代码修改包括几部分： DVR Router network namespace 的创建和删除 DVR Router 相关的 flows DVR Router 的 ARP 表 1.1 DVR Router 相关 network namespace 的创建和删除1.1.1 qrouter 在计算和网络节点上的删除和创建 对于每一个 DVR Router，在每个分布了和 route...
Spark大数据分与实践笔记（第六章 Kafka分布式发布订阅消息系统-04）
Lydia868的专栏
08-29 453
Kafka Streams是Apache Kafka开源项目的一个流处理框架，它是基于Kafka的生产者和消费者,为开发者提供了流式处理的能力，具有低延迟性、高扩展性、弹性、容错的特点，易于集成到现有的应用程序中。Kafka Streams是一套处理分析Kafka中存储数据的客户端类库， 处理完的数据可以重新写回Kafka,也可以发送给外部存储系统。...
OpenVSwitch数据面流表cache机制浅谈
Netfilter
12-12 5999
稍许笔墨写了一篇关于OpenVSwitch(以下简称OVS)的文章： https://blog.csdn.net/dog250/article/details/103492099 但有些事情并没有说清楚。 关于OVS的流表是如何映射成数据平面的Flow cache的，有必要单独写一篇文章来说一下。 下面的Paper描述了整个过程的来龙去脉，不可不读： http://www.openvswitch....
OVS的CT action
greenlaomao的博客
05-15 3935
CT相关内容。1. match部分，ct_state，ct_zone，ct_mark，ct_label2. action部分，     commit，提交一个新的connection给connection tracker模块    table，fork一份pipeline，报文copy一份送给connection tracker，然后从当前指定table重入    zone，制定connectio...
openvswitch 通过ofproto/trace trace跟踪数据包匹配的流表
gyunling的专栏
03-19 3212
目录 1. 解决的问题需求 2. 使用方法 （一）解决的问题需求 我们在使用ovs的时候，通常会遇到虚拟机发出的数据包出现通信不通的情况，而不知道是ovs的哪条流表导致的。此时我们可以通过ovs-appctl ofproto/trace 来跟踪虚拟机发出来的数据包经过的pipeline 路径。 （二）使用方法 命令行语法： ovs-appctl ofproto/trace {[dp_name] odp_flow | bridge br_flow} [OPTIONS...] [-generat
antrea-handson
04-16
安特里亚·汉森 尝试一点点的anni cni功能。 基于版本v0.13.1 安装 wget https://github.com/vmware-tanzu/antrea/releases/download/v0.13.1/antrea.yml 如果要启用功能并将其应用... kubectl apply -f ./antrea.yml
antrea：基于Open vSwitch的Kubernetes网络
02-04
它使用作为网络数据平面，在Layer3 / 4上运行，为Kubernetes集群提供网络和安全服务。 Open vSwitch是一种广泛采用的高性能可编程虚拟交换机。 Antrea利用它来实现Pod网络和安全功能。 例如，Open vSwitch使Antrea...
Antrea对Kubernetes Pod网络的加速和一键诊断.pdf
10-12
Antrea架构包括工作节点、主节点和控制平面数据平面。每个工作节点上运行Antrea代理和kube-proxy，主节点上有Antrea控制器，所有组件通过kubelet与Kubernetes API交互。此外，Antrea使用自定义资源定义（CRDs）来...
Antrea对Kubernetes Pod网络的加速和一键诊断.pptx
10-12
【Antrea：Kubernetes Pod网络加速与一键诊断的利器】 Antrea是一个开源的容器网络插件（CNI），专为Kubernetes集群设计，提供Pod之间的连接性、NetworkPolicy的实施以及ClusterIP的分布式负载均衡。它利用Open ...
安特里亚antrea：基于Open vSwitch的Kubernetes网络
最新发布
06-02
它使用作为网络数据平面，在Layer3 / 4上运行，为Kubernetes集群提供网络和安全服务。 Open vSwitch是一种广泛采用的高性能可编程虚拟交换机。 Antrea利用它来实现Pod网络和安全功能。 例如，Open vSwitch使Antrea...
OVN学习整理
weixin_30747253的博客
07-06 5008
部署OVN网络拓扑 OVN-安装软件包 /etc/yum.repos.d/CentOS-OpenStack-ocata.repo # yum list installed | grep openvswitch openvswitch.x86_64 1:2.9.0-3.el7 @centos-openstack-ocata o...
ovs conntrack及nat
fengcai_ke的博客
07-08 1934
ovs连接跟踪及nat实现分析
Open vSwitch---流表控制主机数据转发实验（三）---实现NAT地址转换（1）
北风
01-22 1782
实验目的：使用NAT转换使得ovs连接的主机可以和外界的互联网连接，实验中，使用114.114.114.114作为ping的对象，NAT转换可以参考前文（https://blog.csdn.net/weixin_40042248/article/details/112534477），本次实验中也是通过iptables来进行nat转换，从而实验和外界进行通信的。 实验原理：具体的拓扑是在上一个实验（https://blog.csdn.net/weixin_40042248/article/details/1
OVS 内核CT实现
榕易的易
08-30 4251
文章目录引言OVS CT 定义CT 匹配域CT 动作TCP 使用范例TCP SYN包TCP SYN-ACK 包TCP ACK 包TCP 断连OVS 内核态 CT 实现CT 实现框架OVS 侧实现 - 进入 CT 前Linux 内核侧实现OVS 侧实现 - 离开 CT 后内核 CT 下的TCP 状态迁移关于 NAT 实现TCP 三次握手之旅SYN 包到来时SYN-ACK 包到来时ACK 包到来时遗留问题参考文档 引言 所有连接跟踪模块的作用，都是在首包到来时，识别并在CT表中生成表项；而当该连接的特定数据包到
软件体系结构风格
qq_51580006的博客
12-06 1651
软件体系结构风格
resubmit和goto区别
bobi的博客
01-01 1290
一、resubmit和goto定义 resubmit官方定义： goto官方定义： 二、resubmit和goto区别 根据官方文档定义我们可知，resubmit和goto都用作在流表间跳转，区别是goto只能跳转到当前table编号之后的流表，而resubmit没有限制 三、resubmit和goto使用场合 在一些场合下只能使用goto ovs-ofctl add-flow anet-br -O OpenFlow13 "table=5,priority=200,ip actions=write_me
一次有趣的 Docker 网络问题排查经历
云原生实验室
07-02 472
前段时间公司的安卓打包服务出现问题，现象是在上传 360 服务器进行加固的时候，非常大概率会卡在上传阶段，长时间重试最后失败。我对这个情况进行了一些排查分析，解决了这个问题，写了这篇长文复...
openstack 网络架构 nova-network + neutron
热门推荐
简单生活
11-16 2万+
openstack网络架构（nova-network/neutron） openstack网络体系中，网络技术没有创新，但用到的技术点非常庞杂，包括bridge、vlan、gre、vxlan、ovs、openflow、sdn、iptables等，当然这里不会做具体技术介绍，概述技术，主要将其与openstack的结合点做详细分析。 nova-network网络架构 在nova-net
Debug OpenvSwitch (by quqi99)
技术并艺术着
12-28 2328
作者：张华 发表于：2020-12-28 版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 GDB调试 sysctl kernel.core_pattern sudo service apport start force_start=1 enabled=1 grep enabled /etc/default/apport sudo killall -SIGSEGV ovs-vswitchd sudo cat /var/log/apport.log sudo apt in
VMware中国团队利用KFServing提升联邦学习模型部署效率
此外，他还参与了Contour（云原生ingress，CNCF孵化项目）、BuildPacks（云原生应用打包，CNCF孵化项目）、Antrea（云原生网络安全，Kubernetes的CNI项目，CNCF沙箱项目）以及Velero（Kubernetes云原生备份和恢复...
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值