openvswitch vxlan 测试

最新推荐文章于 2022-11-29 16:40:29 发布
最新推荐文章于 2022-11-29 16:40:29 发布
阅读量734 收藏
点赞数
分类专栏： openstack 文章标签： openstack openvswitch vxlan 测试
记录学习中的各种问题以及自己的理解，供以后温故以及分享一下，错误之处，希望批评指正，以免误导更多人。如有侵权，请联系删除。谢谢！
本文链接：https://blog.csdn.net/Man_In_The_Night/article/details/108296990
版权

环境： centos7.6, openvsitch 2.11.0

一、创建模拟接口
  • k8s01 节点执行

安装 openvswitch，需要安装 openstack yum 源

yum install -y openvswitch
systemctl start openvswitch

k8s01 创建、配置 vmA1 接口

[root@k8s01 ~]# ip link add dev vmA1-sw type veth peer name vmA1
[root@k8s01 ~]# ip link set vmA1-sw up
[root@k8s01 ~]# ip link set vmA1 up
[root@k8s01 ~]# ip addr add 192.168.60.11/24 dev vmA1

k8s01 创建、配置 vmB1 接口

[root@k8s01 ~]# ip link add dev vmB1-sw type veth peer name vmB1
[root@k8s01 ~]# ip link set vmB1-sw up
[root@k8s01 ~]# ip link set vmB1 up
[root@k8s01 ~]# ip addr add 192.168.70.11/24 dev vmB1

创建 tenantA 和 tenantB 网桥，它们代表虚拟交换机

[root@k8s01 31956]# ovs-vsctl add-br tenantA
[root@k8s01 31956]# ovs-vsctl add-br tenantB

将模拟接口插到网桥上

[root@k8s01 ~]# ovs-vsctl add-port tenantA vmA1-sw
[root@k8s01 ~]# ovs-vsctl add-port tenantB vmB1-sw
[root@k8s01 ~]# ovs-vsctl show
2d3e5812-033a-4641-9064-1268e693c49a
    Bridge tenantA
        Port tenantA
            Interface tenantA
                type: internal
        Port "vmA1-sw"
            Interface "vmA1-sw"
    Bridge tenantB
        Port tenantB
            Interface tenantB
                type: internal
        Port "vmB1-sw"
            Interface "vmB1-sw"
    ovs_version: "2.11.0"
  • k8s02 节点执行

安装 openvswitch，需要安装 openstack yum 源

yum install -y openvswitch
systemctl start openvswitch

k8s02 创建、配置 vmA2 接口

[root@k8s02 ~]# ip link add dev vmA2-sw type veth peer name vmA2
[root@k8s02 ~]# ip link set vmA2-sw up
[root@k8s02 ~]# ip link set vmA2 up
[root@k8s02 ~]# ip addr add 192.168.60.12/24 dev vmA2

k8s02 创建、配置 vmB2接口

[root@k8s02 ~]# ip link add dev vmB2-sw type veth peer name vmB2
[root@k8s02 ~]# ip link set vmB2-sw up
[root@k8s02 ~]# ip link set vmB2 up
[root@k8s02 ~]# ip addr add 192.168.70.12/24 dev vmB2

创建 tenantA 和 tenantB 网桥，它们代表虚拟交换机

[root@k8s02 31956]# ovs-vsctl add-br tenantA
[root@k8s02 31956]# ovs-vsctl add-br tenantB

将模拟接口插到网桥上

[root@k8s02 ~]# ovs-vsctl add-port tenantA vmA2-sw
[root@k8s02 ~]# ovs-vsctl add-port tenantB vmB2-sw
[root@k8s02 ~]# ovs-vsctl show
2d3e5812-033a-4641-9064-1268e693c49a
    Bridge tenantA
        Port tenantA
            Interface tenantA
                type: internal
        Port "vmA2-sw"
            Interface "vmA2-sw"
    Bridge tenantB
        Port tenantB
            Interface tenantB
                type: internal
        Port "vmB2-sw"
            Interface "vmB2-sw"
    ovs_version: "2.11.0"
  • 在 k8s02 上 ping k8s01 模拟接口，不通
[root@k8s02 ~]# ping -c 2 192.168.60.11
PING 192.168.60.11 (192.168.60.11) 56(84) bytes of data.
From 192.168.60.12 icmp_seq=1 Destination Host Unreachable
From 192.168.60.12 icmp_seq=2 Destination Host Unreachable

--- 192.168.60.11 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
pipe 2
[root@k8s02 ~]# ping -c 2 192.168.70.11
PING 192.168.70.11 (192.168.70.11) 56(84) bytes of data.
From 192.168.70.12 icmp_seq=1 Destination Host Unreachable
From 192.168.70.12 icmp_seq=2 Destination Host Unreachable

--- 192.168.70.11 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
pipe 2

注意事项：

1、使用 ovs-vctl 报错

[root@k8s02 ~]# ovs-vsctl add-br tenantA
net_mlx5: cannot load glue library: /lib64/libmlx5.so.1: version `MLX5_1.6' not found (required by /usr/lib64/dpdk-pmds-glue/librte_pmd_mlx5_glue.so.18.11.0)
net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)

解决方法

yum install libmlx5
二、创造 vm 之间的 vxlan 管道
  • k8s01 节点执行
[root@k8s01 ~]# ovs-vsctl add-port tenantA vxlanA -- set interface vxlanA type=vxlan options:remote_ip=10.2.7.201 options:key=5000
[root@k8s01 ~]# ovs-vsctl add-port tenantB vxlanB -- set interface vxlanB type=vxlan options:remote_ip=10.2.7.201 options:key=6000
[root@k8s01 ~]# ovs-vsctl show
2d3e5812-033a-4641-9064-1268e693c49a
    Bridge tenantA
        Port tenantA
            Interface tenantA
                type: internal
        Port "vmA1-sw"
            Interface "vmA1-sw"
        Port vxlanA
            Interface vxlanA
                type: vxlan
                options: {key="5000", remote_ip="10.2.7.201"}
    Bridge tenantB
        Port tenantB
            Interface tenantB
                type: internal
        Port vxlanB
            Interface vxlanB
                type: vxlan
                options: {key="6000", remote_ip="10.2.7.201"}
        Port "vmB1-sw"
            Interface "vmB1-sw"
    ovs_version: "2.11.0"
  • k8s02 节点执行
[root@k8s02 ~]# ovs-vsctl add-port tenantA vxlanA -- set interface vxlanA type=vxlan options:remote_ip=10.2.7.200 options:key=5000
[root@k8s02 ~]# ovs-vsctl add-port tenantB vxlanB -- set interface vxlanB type=vxlan options:remote_ip=10.2.7.200 options:key=6000
[root@k8s02 ~]#  ovs-vsctl show
c3b22dc3-7884-41cc-98c8-da60451491c3
    Bridge tenantA
        Port vxlanA
            Interface vxlanA
                type: vxlan
                options: {key="5000", remote_ip="10.2.7.200"}
        Port tenantA
            Interface tenantA
                type: internal
        Port "vmA2-sw"
            Interface "vmA2-sw"
    Bridge tenantB
        Port vxlanB
            Interface vxlanB
                type: vxlan
                options: {key="6000", remote_ip="10.2.7.200"}
        Port "vmB2-sw"
            Interface "vmB2-sw"
        Port tenantB
            Interface tenantB
                type: internal
    ovs_version: "2.11.0"
  • 在 k8s02 上 ping k8s01 模拟接口，能通
[root@k8s02 ~]# ping -c 2 192.168.60.11
PING 192.168.60.11 (192.168.60.11) 56(84) bytes of data.
64 bytes from 192.168.60.11: icmp_seq=1 ttl=64 time=1.81 ms
64 bytes from 192.168.60.11: icmp_seq=2 ttl=64 time=0.279 ms

--- 192.168.60.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.279/1.048/1.818/0.770 ms
[root@k8s02 ~]# ping -c 2 192.168.70.11
PING 192.168.70.11 (192.168.70.11) 56(84) bytes of data.
64 bytes from 192.168.70.11: icmp_seq=1 ttl=64 time=1.21 ms
64 bytes from 192.168.70.11: icmp_seq=2 ttl=64 time=0.204 ms

--- 192.168.70.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.204/0.710/1.217/0.507 ms
三、openvswitch 控制命令

ovs-vsctl 是配置 openvswitch 最重要的命令，ovs-vsctl 与 ovsdb-server 通信，ovsdb-server 管理着 openvswitch 的配置信息数据库。

ovs-vsctl 一次可以执行一条或多条命令，多条命令用 “–” 分开

帮助信息：ovs-vsctl --help 、 man 5 ovs-vswitchd.conf.db、ovs-vsctl list

显示已存在网桥的 db 信息

[root@k8s01 ~]#  ovs-vsctl list bridge
_uuid               : ed1cf240-72b2-4d0d-80ba-34fe4319805f
auto_attach         : []
controller          : []
datapath_id         : "000042f21ced0d4d"
datapath_type       : ""
datapath_version    : "<unknown>"
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : tenantB
netflow             : []
other_config        : {}
ports               : [05e579af-f158-4a27-b046-3d45a63f463f, a33d043c-2a82-4f52-8c10-4a28eab38345, b17ade3a-31bf-4cd2-8f1c-8819b82c53aa]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

_uuid               : af7cbcf0-95c4-4f3e-a89a-6eb5a2a44154
auto_attach         : []
controller          : []
datapath_id         : "0000f2bc7caf3e4f"
datapath_type       : ""
datapath_version    : "<unknown>"
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : tenantA
netflow             : []
other_config        : {}
ports               : [a631ac6c-6039-4171-812a-aefc24487ac0, f26cfa5d-fd35-439a-a934-86bdfe632d5e, f87dd940-1d71-437a-8694-a003c25c7c7b]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false

显示网桥 tenanaA 的 db 信息

[root@k8s01 ~]# ovs-vsctl list bridge tenantA
_uuid               : af7cbcf0-95c4-4f3e-a89a-6eb5a2a44154
auto_attach         : []
controller          : []
datapath_id         : "0000f2bc7caf3e4f"
datapath_type       : ""
datapath_version    : "<unknown>"
external_ids        : {}
fail_mode           : []
flood_vlans         : []
flow_tables         : {}
ipfix               : []
mcast_snooping_enable: false
mirrors             : []
name                : tenantA
netflow             : []
other_config        : {}
ports               : [a631ac6c-6039-4171-812a-aefc24487ac0, f26cfa5d-fd35-439a-a934-86bdfe632d5e, f87dd940-1d71-437a-8694-a003c25c7c7b]
protocols           : []
rstp_enable         : false
rstp_status         : {}
sflow               : []
status              : {}
stp_enable          : false
四、openvswitch port

传统的交换机端口是 2 层端口，流量在该交换机端口之间传输，这些端口没有 3 层的 ip 配置。即使是的 linux bridge 也能发现这些特性，比如 eth0 配置了 ip，再把它加入到一个 bridge，那么你将失去 eth0 的连接，因为 eth0 此时只作为 2 层端口，可以将 ip 移到连接 eth0 的 bridge 接口（比如 br0）

internal port

openvswith 用 internal port 提供一个解决方法。 internal port 是一个 3 层端口，能够暴露到 openvswitch 外面，所以能够进行 ip 设置

k8s01 上配置

[root@k8s01 ~]#  ovs-vsctl add-port tenantA internalPort -- set interface internalPort type=internal
[root@k8s01 ~]# ip addr add 192.168.60.50/24 dev internalPort
[root@k8s01 ~]# ip link set internalPort up

k8s02 上 ping 192.168.60.50

[root@k8s02 ~]#  ping -c2 192.168.60.50
PING 192.168.60.50 (192.168.60.50) 56(84) bytes of data.
64 bytes from 192.168.60.50: icmp_seq=1 ttl=64 time=1.36 ms
64 bytes from 192.168.60.50: icmp_seq=2 ttl=64 time=0.162 ms

--- 192.168.60.50 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.162/0.763/1.364/0.601 ms

internal port 的名称就像网桥名称一样

[root@k8s01 ~]# ovs-vsctl show
2d3e5812-033a-4641-9064-1268e693c49a
    Bridge tenantA
        Port tenantA
            Interface tenantA
                type: internal
        Port internalPort
            Interface internalPort
                type: internal
        Port "vmA1-sw"
            Interface "vmA1-sw"
        Port vxlanA
            Interface vxlanA
                type: vxlan
                options: {key="5000", remote_ip="10.2.7.201"}
    Bridge tenantB
        Port tenantB
            Interface tenantB
                type: internal
        Port vxlanB
            Interface vxlanB
                type: vxlan
                options: {key="6000", remote_ip="10.2.7.201"}
        Port "vmB1-sw"
            Interface "vmB1-sw"
    ovs_version: "2.11.0
[root@k8s01 ~]# ip link | grep tenant
197: tenantA: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
198: tenantB: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
mirror port

交换机有 mirror port 和 span prot，该 port 可以作为特定流量的镜像，用于拍错。 openvswith 也支持 mirror port

创建一个内部的 mirror port，我们需要一个端口并在上面进行 wireshark/tcpdump 监听

[root@k8s01 ~]# ovs-vsctl add-port tenantA mirrorPort -- set interface mirrorPort type=internal

创建 mirror 配置

[root@k8s01 ~]# ovs-vsctl --id=@vmA1-sw get port vmA1-sw --\
>                           --id=@mirrorPort get port mirrorPort --\
>                           --id=@mirror create mirror name=mirror \
>                           select-dst-port=@vmA1-sw select-src-port=@vmA1-sw output-port=@mirrorPort --\
>                           set bridge tenantA mirrors=@mirror
8d73fe4f-b111-4f8f-a3d1-b8672dbe037a

安装 wireshark

[root@k8s01 ~]# yum -y install wireshark

k8s02 ping

[root@k8s02 ~]#  ping  192.168.60.11
PING 192.168.60.11 (192.168.60.11) 56(84) bytes of data.
64 bytes from 192.168.60.11: icmp_seq=1 ttl=64 time=0.966 ms
64 bytes from 192.168.60.11: icmp_seq=2 ttl=64 time=0.211 ms
64 bytes from 192.168.60.11: icmp_seq=3 ttl=64 time=0.193 ms
64 bytes from 192.168.60.11: icmp_seq=4 ttl=64 time=0.235 ms
64 bytes from 192.168.60.11: icmp_seq=5 ttl=64 time=0.277 ms
...

监听 mirror 端口

[root@k8s01 ~]# ip link set mirrorPort up
[root@k8s01 ~]# tshark -c 6 -i mirrorPort
Running as user "root" and group "root". This could be dangerous.
Capturing on 'mirrorPort'
  1 0.000000000 192.168.60.12 -> 192.168.60.11 ICMP 98 Echo (ping) request  id=0x7aec, seq=1/256, ttl=64
  2 0.000140651 192.168.60.11 -> 192.168.60.12 ICMP 98 Echo (ping) reply    id=0x7aec, seq=1/256, ttl=64 (request in 1)
  3 0.999594777 192.168.60.12 -> 192.168.60.11 ICMP 98 Echo (ping) request  id=0x7aec, seq=2/512, ttl=64
  4 0.999626499 192.168.60.11 -> 192.168.60.12 ICMP 98 Echo (ping) reply    id=0x7aec, seq=2/512, ttl=64 (request in 3)
  5 1.999676670 192.168.60.12 -> 192.168.60.11 ICMP 98 Echo (ping) request  id=0x7aec, seq=3/768, ttl=64
  6 1.999707851 192.168.60.11 -> 192.168.60.12 ICMP 98 Echo (ping) reply    id=0x7aec, seq=3/768, ttl=64 (request in 5)
6 packets captured

监听 eth0 端口，发现 192.168.60.12 > 192.168.60.11 流量是封装成 vxlan 通过 eth0 发送

[root@k8s01 ~]# tcpdump -i eth0 host 10.2.7.201 -n | grep -C2 192.168.60.11
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:09:44.905533 IP 10.2.7.200.51404 > 10.2.7.201.2380: Flags [.], ack 1949, win 1424, options [nop,nop,TS val 2565202554 ecr 1863325088], length 0
09:09:44.906178 IP 10.2.7.201.53192 > 10.2.7.200.4789: VXLAN, flags [I] (0x08), vni 5000
IP 192.168.60.12 > 192.168.60.11: ICMP echo request, id 31344, seq 9, length 64
09:09:44.906281 IP 10.2.7.200.39328 > 10.2.7.201.4789: VXLAN, flags [I] (0x08), vni 5000
IP 192.168.60.11 > 192.168.60.12: ICMP echo reply, id 31344, seq 9, length 64
09:09:44.906463 IP 10.2.7.201.ssh > 10.2.7.107.55590: Flags [P.], seq 3157118280:3157118380, ack 17960795, win 386, options [nop,nop,TS val 1863325089 ecr 4229146211], length 100
09:09:44.906569 IP 10.2.7.107.55590 > 10.2.7.201.ssh: Flags [.], ack 100, win 1424, options [nop,nop,TS val 4229147211 ecr 1863325089], length 0
--
...

清理 mirror

[root@k8s01 ~]# ovs-vsctl clear bridge tenantA mirrors
[root@k8s01 ~]# ovs-vsctl del-port mirrorPort

注意：mirror 设置中 --id 命令提供接口的别名，需要接口的 id 而不是名称。不能给 patch ports 创建 mirror，如果需要 mirror path ports 流量，请使用 veth pairs。

patch port

patch ports 就像连接两个交换机之间的电缆，或者插入 openvswitch 网桥的 veth pair

创建 tenantC 网桥以及设置 tenantC internal port

k8s01 节点

[root@k8s01 ~]# ovs-vsctl add-br tenantC
[root@k8s01 ~]# ip addr add 192.168.80.11/24 dev tenantC
[root@k8s01 ~]# ip link set tenantC up

k8s02 节点

[root@k8s02 ~]# ovs-vsctl add-br tenantC
[root@k8s02 ~]# ip addr add 192.168.80.12/24 dev tenantC
[root@k8s02 ~]# ip link set tenantC up

k8s02 ping 192.168.80.11，没有 patch 连接，所以不通

[root@k8s02 ~]#  ping -c2 192.168.80.11
PING 192.168.80.11 (192.168.80.11) 56(84) bytes of data.
From 192.168.80.12 icmp_seq=1 Destination Host Unreachable
From 192.168.80.12 icmp_seq=2 Destination Host Unreachable

--- 192.168.80.11 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
pipe 2

建立 path 连接

k8s01 节点

[root@k8s01 ~]# ovs-vsctl add-port tenantC patchC --\
>                           add-port tenantA patchA --\
>                           set interface patchC type=patch options:peer=patchA --\
>                           set interface patchA type=patch options:peer=patchC

k8s02 节点

[root@k8s02 ~]#  ovs-vsctl add-port tenantC patchC --\
>                           add-port tenantA patchA --\
>                           set interface patchC type=patch options:peer=patchA --\
>                           set interface patchA type=patch options:peer=patchC

k8s02 ping 192.168.80.11

[root@k8s02 ~]# ping -c2 192.168.80.11
PING 192.168.80.11 (192.168.80.11) 56(84) bytes of data.
64 bytes from 192.168.80.11: icmp_seq=1 ttl=64 time=1.47 ms
64 bytes from 192.168.80.11: icmp_seq=2 ttl=64 time=0.189 ms

--- 192.168.80.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.189/0.833/1.478/0.645 ms
vlans

k8s01 节点

[root@k8s01 ~]# ovs-vsctl add-port tenantC vlanPortC tag=10 --\
>                 set interface vlanPortC type=internal
[root@k8s01 ~]# ip addr add 192.168.90.11/24 dev vlanPortC
[root@k8s01 ~]# ip link set vlanPortC up

k8s02 节点

[root@k8s02 ~]#  ovs-vsctl add-port tenantC vlanPortC tag=10 --\
>                 set interface vlanPortC type=internal
[root@k8s02 ~]# ip addr add 192.168.90.12/24 dev vlanPortC
[root@k8s02 ~]#  ip link set vlanPortC up

k8s02 ping 192.168.90.11

[root@k8s02 ~]# ping -c2 192.168.90.11
PING 192.168.90.11 (192.168.90.11) 56(84) bytes of data.
64 bytes from 192.168.90.11: icmp_seq=1 ttl=64 time=1.67 ms
64 bytes from 192.168.90.11: icmp_seq=2 ttl=64 time=0.178 ms

--- 192.168.90.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.178/0.927/1.676/0.749 ms

监听 eth0 端口，发现 192.168.90.12 > 192.168.90.11 流量是封装成 vxlan 通过 eth0 发送

[root@k8s01 ~]# tcpdump -i eth0 -n| grep -C2 192.168.90.12
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:28:34.066375 IP 10.2.7.202.46040 > 10.2.7.200.2380: Flags [.], ack 886, win 1424, options [nop,nop,TS val 1172533648 ecr 2569931715], length 0
10:28:34.068319 IP 10.2.7.201.42349 > 10.2.7.200.4789: VXLAN, flags [I] (0x08), vni 5000
IP 192.168.90.12 > 192.168.90.11: ICMP echo request, id 19832, seq 134, length 64
10:28:34.068389 IP 10.2.7.200.49438 > 10.2.7.201.4789: VXLAN, flags [I] (0x08), vni 5000
IP 192.168.90.11 > 192.168.90.12: ICMP echo reply, id 19832, seq 134, length 64
10:28:34.068592 IP 10.2.7.201.ssh > 10.2.7.107.55590: Flags [P.], seq 3157289604:3157289704, ack 17976739, win 431, options [nop,nop,TS val 1868054235 ecr 4233875337], length 100
..
vxlan 和 gre ports

vxlan 和 gre 很相似。移除 tenantA 和 tenantC 之间的 patch，gre 管道将处理 tenanC 网桥之间的流量信息

删除 patch ports

k8s01 节点

k8s01 节点[root@k8s01 ~]# ovs-vsctl del-port patchC
[root@k8s01 ~]# ovs-vsctl del-port patchA

k8s02 节点

[root@k8s02 ~]# ovs-vsctl del-port patchC
[root@k8s02 ~]# ovs-vsctl del-port patchA

k8s02 ping 192.168.90.11，不通

[root@k8s02 ~]#  ping -c2 192.168.90.11
PING 192.168.90.11 (192.168.90.11) 56(84) bytes of data.
From 192.168.90.12 icmp_seq=1 Destination Host Unreachable
From 192.168.90.12 icmp_seq=2 Destination Host Unreachable

--- 192.168.90.11 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms
pipe 2

增加 gre 管道

k8s01 节点

[root@k8s01 ~]# ovs-vsctl add-port tenantC greC -- set interface greC type=gre options:remote_ip=10.2.7.201

k8s02 节点

[root@k8s02 ~]# ovs-vsctl add-port tenantC greC -- set interface greC type=gre options:remote_ip=10.2.7.200

k8s02 ping 192.168.90.11

[root@k8s02 ~]# ping -c2 192.168.90.11
PING 192.168.90.11 (192.168.90.11) 56(84) bytes of data.
64 bytes from 192.168.90.11: icmp_seq=1 ttl=64 time=1.28 ms
64 bytes from 192.168.90.11: icmp_seq=2 ttl=64 time=0.173 ms

--- 192.168.90.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.173/0.729/1.286/0.557 ms

监听 eth0 端口，发现 192.168.90.12 > 192.168.90.11 流量是封装成 GREv0 通过 eth0 发送

[root@k8s01 ~]# tcpdump  -n -i eth0 | grep -C2 192.168.90.12
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:50:38.842380 IP 10.2.7.201.ssh > 10.2.7.107.55590: Flags [P.], seq 1988:2080, ack 393, win 431, options [nop,nop,TS val 1869379008 ecr 4235201072], length 92
10:50:38.842472 IP 10.2.7.107.55590 > 10.2.7.201.ssh: Flags [.], ack 2080, win 1424, options [nop,nop,TS val 4235201101 ecr 1869379008], length 0
10:50:38.842498 IP 10.2.7.201 > 10.2.7.200: GREv0, length 106: IP 192.168.90.12 > 192.168.90.11: ICMP echo request, id 3857, seq 1, length 64
10:50:38.842607 IP 10.2.7.107.ssh > 10.3.57.11.34344: Flags [P.], seq 2640:2752, ack 641, win 399, options [nop,nop,TS val 4235201101 ecr 3872193733], length 112
10:50:38.842731 IP 10.3.57.11.34344 > 10.2.7.107.ssh: Flags [.], ack 2752, win 2440, options [nop,nop,TS val 3872193762 ecr 4235201101], length 0
10:50:38.842902 IP 10.2.7.200 > 10.2.7.201: GREv0, length 106: IP 192.168.90.11 > 192.168.90.12: ICMP echo reply, id 3857, seq 1, length 64
10:50:38.843231 IP 10.2.7.201.ssh > 10.2.7.107.55590: Flags [P.], seq 2080:2180, ack 393, win 431, options [nop,nop,TS val 1869379009 ecr 4235201101], length 100
10:50:38.843267 IP 10.2.7.107.55590 > 10.2.7.201.ssh: Flags [.], ack 2180, win 1424, options [nop,nop,TS val 4235201101 ecr 1869379009], length 0
--
...

监听 gre_sys 端口，发现 192.168.90.12 > 192.168.90.11 会通过 gre_sys 接口

[root@k8s01 ~]# tcpdump  -n -i gre_sys 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gre_sys, link-type EN10MB (Ethernet), capture size 262144 bytes
10:49:28.236776 IP 192.168.80.12 > 192.168.80.11: ICMP echo request, id 2707, seq 36, length 64
10:49:28.236827 IP 192.168.80.11 > 192.168.80.12: ICMP echo reply, id 2707, seq 36, length 64
10:49:29.236759 IP 192.168.80.12 > 192.168.80.11: ICMP echo request, id 2707, seq 37, length 64
10:49:29.236809 IP 192.168.80.11 > 192.168.80.12: ICMP echo reply, id 2707, seq 37, length 64
10:49:30.236728 IP 192.168.80.12 > 192.168.80.11: ICMP echo request, id 2707, seq 38, length 64
10:49:30.236773 IP 192.168.80.11 > 192.168.80.12: ICMP echo reply, id 2707, seq 38, length 64
...
五、mtu 问题

mtu 最大传输单元，一般最大值为 1500，超过就会切分。vxlan 使用封装技术，传输单元可能会超过 底层网络最大传输单元 1500，这回导致问题，比如下面。

k8s02 作为server

[root@k8s02 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.70.11, port 40230
[  5] local 192.168.70.12 port 5201 connected to 192.168.70.11 port 40232
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec                  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  
[  5]  10.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
...

k8s01 作为 client

[root@k8s01 ~]# iperf3 -c 192.168.70.12
Connecting to host 192.168.70.12, port 5201
[  4] local 192.168.70.11 port 40232 connected to 192.168.70.12 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  84.8 KBytes   694 Kbits/sec    2   1.41 KBytes       
[  4]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  4]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  4]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  4]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  4]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  84.8 KBytes  69.5 Kbits/sec    5             sender
[  4]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.
  • 解决方法 1

为了避免这样的为题，必须调整底层网络的 mtu。通过计算额外的 header，VXLAN + UDP + IP + Ethernet，得到 mtu 1554

[root@k8s02 ~]# ip link set eth0 mtu 1554
[root@k8s02 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.70.11, port 45934
[  5] local 192.168.70.12 port 5201 connected to 192.168.70.11 port 45936
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   316 MBytes  2.65 Gbits/sec                  
[  5]   1.00-2.00   sec   329 MBytes  2.76 Gbits/sec                  
[  5]   2.00-3.00   sec   356 MBytes  2.99 Gbits/sec                  
[  5]   3.00-4.00   sec   353 MBytes  2.96 Gbits/sec                  
[  5]   4.00-5.00   sec   353 MBytes  2.96 Gbits/sec                  
[  5]   5.00-6.00   sec   344 MBytes  2.88 Gbits/sec                  
[  5]   6.00-7.00   sec   348 MBytes  2.92 Gbits/sec                  
[  5]   7.00-8.00   sec   348 MBytes  2.92 Gbits/sec                  
[  5]   8.00-9.00   sec   342 MBytes  2.87 Gbits/sec                  
[  5]   9.00-10.00  sec   344 MBytes  2.89 Gbits/sec                  
[  5]  10.00-10.04  sec  15.1 MBytes  2.99 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  3.37 GBytes  2.88 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

[root@k8s01 ~]# ip link set eth0 mtu 1554
[root@k8s01 ~]# iperf3 -c 192.168.70.12
Connecting to host 192.168.70.12, port 5201
[  4] local 192.168.70.11 port 45936 connected to 192.168.70.12 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   332 MBytes  2.79 Gbits/sec    8    730 KBytes       
[  4]   1.00-2.00   sec   330 MBytes  2.77 Gbits/sec    0   1022 KBytes       
[  4]   2.00-3.00   sec   356 MBytes  2.99 Gbits/sec    0   1.23 MBytes       
[  4]   3.00-4.00   sec   352 MBytes  2.96 Gbits/sec   32   1.42 MBytes       
[  4]   4.00-5.00   sec   352 MBytes  2.96 Gbits/sec    0   1.59 MBytes       
[  4]   5.00-6.00   sec   345 MBytes  2.89 Gbits/sec   14   1.30 MBytes       
[  4]   6.00-7.00   sec   348 MBytes  2.92 Gbits/sec    0   1.49 MBytes       
[  4]   7.00-8.00   sec   349 MBytes  2.93 Gbits/sec    9   1.19 MBytes       
[  4]   8.00-9.00   sec   341 MBytes  2.86 Gbits/sec    0   1.39 MBytes       
[  4]   9.00-10.00  sec   345 MBytes  2.90 Gbits/sec   32   1.55 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.37 GBytes  2.89 Gbits/sec   95             sender
[  4]   0.00-10.00  sec  3.37 GBytes  2.89 Gbits/sec                  receiver

iperf Done.
  • 解决方法 2

也可以减少 vmB1 的 mtu

[root@k8s02 ~]# ip link set vmB2 mtu 1450
[root@k8s02 ~]# ip link set vmB2-sw mtu 1450
[root@k8s02 ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.70.11, port 52206
[  5] local 192.168.70.12 port 5201 connected to 192.168.70.11 port 52208
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   298 MBytes  2.50 Gbits/sec                  
[  5]   1.00-2.00   sec   326 MBytes  2.73 Gbits/sec                  
[  5]   2.00-3.00   sec   327 MBytes  2.75 Gbits/sec                  
[  5]   3.00-4.00   sec   323 MBytes  2.71 Gbits/sec                  
[  5]   4.00-5.00   sec   334 MBytes  2.80 Gbits/sec                  
[  5]   5.00-6.00   sec   341 MBytes  2.86 Gbits/sec                  
[  5]   6.00-7.00   sec   342 MBytes  2.87 Gbits/sec                  
[  5]   7.00-8.00   sec   344 MBytes  2.89 Gbits/sec                  
[  5]   8.00-9.00   sec   345 MBytes  2.89 Gbits/sec                  
[  5]   9.00-10.00  sec   341 MBytes  2.86 Gbits/sec                  
[  5]  10.00-10.04  sec  13.9 MBytes  2.86 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  3.26 GBytes  2.79 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

[root@k8s01 ~]# ip link set vmA1 mtu 1450
[root@k8s01 ~]# ip link set vmA1-sw mtu 1450
[root@k8s01 ~]# iperf3 -c 192.168.70.12
Connecting to host 192.168.70.12, port 5201
[  4] local 192.168.70.11 port 52208 connected to 192.168.70.12 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   312 MBytes  2.61 Gbits/sec   47    754 KBytes       
[  4]   1.00-2.00   sec   328 MBytes  2.75 Gbits/sec    0   1.00 MBytes       
[  4]   2.00-3.00   sec   327 MBytes  2.75 Gbits/sec    0   1.21 MBytes       
[  4]   3.00-4.00   sec   322 MBytes  2.71 Gbits/sec    0   1.39 MBytes       
[  4]   4.00-5.00   sec   334 MBytes  2.80 Gbits/sec    0   1.55 MBytes       
[  4]   5.00-6.00   sec   341 MBytes  2.86 Gbits/sec   23   1.28 MBytes       
[  4]   6.00-7.00   sec   342 MBytes  2.87 Gbits/sec   33   1.45 MBytes       
[  4]   7.00-8.00   sec   345 MBytes  2.89 Gbits/sec    1   1.17 MBytes       
[  4]   8.00-9.00   sec   344 MBytes  2.88 Gbits/sec    0   1.37 MBytes       
[  4]   9.00-10.00  sec   341 MBytes  2.86 Gbits/sec    0   1.53 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec  104             sender
[  4]   0.00-10.00  sec  3.26 GBytes  2.80 Gbits/sec                  receiver

iperf Done.

参考文章：

OpenStack Networking: Open vSwitch and VXLAN introduction

Man_In_The_Night
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
两台虚拟服务器如何级联,[教程] 利用open vswitch建立vxlan隧道实现不同主机上的虚拟交换机级联...
weixin_36083434的博客
08-12 808
写在开头在某些环境下，需要实现两台物理机中的openvswitch交换机级联，以实现两台交换机中的设备互相通讯，这里使用vxlan隧道技术，将数据包封装在UDP中，通过以太网实现数据包传输。VXLAN是一种大二层的虚拟技术，主要的技术原理是引入了一个UDP外层隧道，作为数据的链路层，而原有的数据报文内容作为隧道净荷来传输，由于外层采用了UDP作为传出手段，就可以让净荷数据轻而易举的在二三层网络中传...
搭建基于Open vSwitch的VxLAN隧道实验
jincm的专栏
05-21 2578
搭建基于Open vSwitch的VxLAN隧道实验 偶然左岸 • 14-12-19 • 3,620 人围观 1. VXLAN简介 VXLAN 是 Virtual eXtensible LANs 的缩写，它是对 VLAN 的一个扩展，是非常新的一个 tunnel 技术，在Open vSwitch中应用也非常多。Linux 内核的 upstream
OpenvSwitch VxLAN实验
陈海峰的博客
11-30 2558
操作系统centos 7.4升级内核 http://blog.csdn.net/chenhaifeng2016/article/details/78675531 安装openvswitch http://blog.csdn.net/chenhaifeng2016/article/details/78674223 未完待续
Configuring VXLAN and GRE Tunnels on OpenvSwitch
云中的虚拟化世界
05-05 3857
CONFIGURING VXLAN AND GRE TUNNELS ON OPENVSWITCH UPDATE: VXLAN ENCAPSULATION VXLan is now upstreamed into the Master build. It is worth mentioning that is the encapsulation only, not the Multi
Openvswitch+vxlan(转载自https://blog.csdn.net/dylloveyou/article/details/72639187)
cisco_eigrp的博客
09-10 1246
本篇文章介绍如何规划及创建openvswitch+vxlan网络，实现实例间及实例与外部的通讯。读者应对OpenStack及其网络服务Neutron有初步的了解。 规划网络 部署节点为一个controller节点(包含网络节点)，两个compute节点。controller节点有3个网卡，分别为eth0(管理和API网络，CIDR为192.168.128.0/24)、eth1(租户网络，CID...
openvswitch-2.5.0-2.el7.x86_64.rpm
09-28
主要用于openstack的网络环境搭建，可以支持VXLAN，gre,本软件在实际工作中测试过，没问题
vxlan-gbp:VXLAN 组策略扩展
07-02
快速通道：流浪运行vagrant up将为两个名为kernel1和kernel2基于 Fedora 20 的 VM 提供启用 VXLAN-GBP 的内核、iproute2 和 Open vSwitch。 使用vagrant ssh登录虚拟机。 这将为您提供以下拓扑： +-----------------...
147-创建 vxlan 并部署 instance1
08-08
在下一节中，我们将深入探讨Open vSwitch如何利用VXLAN技术，实现虚拟机间的通信，并了解其在网络层和数据链路层的具体操作。这将涉及VXLAN接口的创建、VXLAN隧道的建立以及数据包在VXLAN网络中的转发过程。通过这些...
ovs-helper
02-05
Open vSwitch是一种开源的虚拟交换机，广泛应用于云计算环境和数据中心，支持多种网络协议和服务，如VXLAN、GRE等。在SDN架构中，OVS通常作为数据平面的一部分，通过OVSDB（Open vSwitch Database）进行配置和管理。...
6WIND-OVS-Demo-Solution-Brief.pdf
01-29
6WIND是一家专注于网络性能优化的软件提供商，其产品6WINDGate™是一款专门用于提升标准Open vSwitch (OVS)性能的包处理软件。这款解决方案旨在解决数据中心中关键的网络性能挑战。通过6WINDGate，OVS在应用服务器...
ovs测试vlan
Flytiger
07-24 2672
ovs端口默认是trunk模式，且所有的VLAN tag都可以通过。 单个网桥实现vlan隔离的场景 实验拓扑如下： 1、首先创建一个网桥ovs-switch，在命名空间里创建两个接口p0和p1。分别分配地址为192.168.1.100/24和192.168.1.101/24 创建网桥并添加接口 ovs-vsctl add-br ovs-switch ovs-vsctl add-port ovs-switch p0 – set Interface p0 ofport_request=100 ovs-vs
openvswitch实践三 vxlan内核态实现与openvswitch实现对比
魏言华的博客
08-11 616
1.vxlan内核态实现 1.1 测试环境 1.2 试验配置 host 172.16.0.100 brctl addbr br0 ip addr add 10.1.1.2/24 dev br0 ip link set dev br0 up ip link add vxlan0 type vxlan id 1 group 239.1.1.1 dev eth1 dstport 4789 brctl addif br0 vxlan0 ip link set dev vxlan0 up ..
通过openvswitch建立vxlan通道&&通过iperf3测速
weixin_30764771的博客
08-21 236
一、建立vxlan通道 系统环境centos7.3，安装完毕openvswitch。 环境说明：两台虚拟机，VM1地址 192.168.211.136 ， VM2地址 192.168.211.183 VM1上的操作： 1、添加网桥 ovs-vsctl add-br br0 2、建立添加端口，建立到vm2网卡的vxlan通道 ovs-vsctl add-port br0 vx1 -- ...
Open vSwith模拟网关实现不同子网的互通
weixin_33717117的博客
08-01 509
一.实验目的 本实验通过Mininet构建子网，并使得OVS作为网关，来模拟子网间的互通。在实验的过程中，我们来学习一下内容：OVS构建子网过程。 OVS设置网关过程。 OVS配置流表过程。 二.实验准备 实验环境我们使用Mininet进行构建，建议到Mininet官方下载最新的Mininet虚拟机，本实验中虚拟机版本是mininet-2.2.1-15...
OVS测试
sun172270102的博客
10-09 1195
Testing 使用open vswitch和各种第三方提供的工具来测试open vswitch。 内置工具 open vswitch提供各种不同的测试套件和其他测试工具来验证ovs的基础功能。执行测试前必须要参照官方文档编译、配置和启动open vswitch。执行测试用例时不需要安装open vswitch或加载、编译内核模块，也不需要root权限。 单元测试 Open vswitch包含了一套内部测试。提交patch之前，建议执行这些内部测试用...
ENSPLAB笔记：配置VXLAN（分布式网关，BGP EVPN方式）（Part2)
googleccsdn的博客
03-19 2412
(继续上一篇的5、结果验证）
深入解析Linux 常用命令--ping
juyin2015的博客
12-20 680
深入解析Linux常用命令–ping深入解析Linux常用命令ping 概述 各参数含义 1.概述ping常用于检测网络状况。2.各参数含义-c count，当ping到一定数量后终止ping。-i interval ，ping包间隔-I interface address，网卡,IP所在网卡ping出-s packetsize，包大小，包含8字节icmp头-S 指定源地
openstack ssh远程连接kvm ：net_mlx5: cannot load glue library: libibverbs.so.1
Futures_的博客
11-21 1896
部署openstack，ssh连接到kvm提示： net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory net_mlx5: cannot initialize PMD due to missing run-time depende...
Ubuntu 20.04 上学习Open vSwitch ：VxLAN
最新发布
哈哈虎的博客
11-29 1492
Ubuntu 20.04 上学习Open vSwitch ：VxLAN
OpenvSwitch安装与配置详解
"未来网络中OpenvSwitch的安装与配置" OpenvSwitch是一个高度可扩展且功能丰富的虚拟交换机解决方案，适用于构建大规模网络自动化环境。它由Nicira Networks开发，并遵循Apache 2.0开源许可证，其核心目标是提供一...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值