记一次bond引起的网络故障

本案中3个关键服务器
物理服务器：192.168.6.63，简称P，(Physical server)
KVM-VM：192.168.6.150，是物理服务器P上的一个KVM虚机，简称VM
NAS：外部NAS服务器，用来做ping/arp测试服务器，简称NAS

物理服务器P的配置：

#uname -a
Linux cz63 4.15.18-11-pve #1 SMP PVE 4.15.18-34 (Mon, 25 Feb 2019 14:51:06 +0100) x86_64 GNU/Linux

#cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp3s0f0
iface enp3s0f0 inet manual

auto enp3s0f1
iface enp3s0f1 inet manual

auto ens1f0
iface ens1f0 inet manual

auto ens1f1
iface ens1f1 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves enp3s0f0 enp3s0f1
    bond-miimon 100
    bond-mode balance-rr

auto bond1
iface bond1 inet manual
    bond-slaves ens1f0 ens1f1
    bond-miimon 100
    bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
    address  192.168.6.63
    netmask  255.255.255.0
    gateway  192.168.6.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet static
    address  10.1.1.63
    netmask  255.255.255.0
    bridge-ports bond1
    bridge-stp off
    bridge-fd 0
#brctl show
bridge name bridge id       STP enabled interfaces
vmbr0       8000.ac1f6b342094   no      bond0
                            tap401000001i0
vmbr1       8000.74a4b500e768   no      bond1

安装KVM-VM，之后VM与外部服务器的连接非常不稳定，90%情况下会出现ping不可达。这里测试了centos、ubuntu、win7都是一样的不稳定。
在VM内执行ping 192.168.6.40（NAS）
结果ping不通。
此时，在P上执行
tcpdump -leni vmbr0 arp
tcpdump -leni tap401000001i0 arp
发现情况如下：

P上
#tcpdump -leni vmbr0 arp | grep 2a:f0:5f:ae:c9:8b
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 > 2a:f0:5f:ae:c9:8b, ethertype ARP (0x0806), length 60: Reply 192.168.6.40 is-at 00:50:56:87:86:b9, length 46
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 28
2a:f0:5f:ae:c9:8b > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.6.40 (ff:ff:ff:ff:ff:ff) tell 192.168.6.150, length 46
00:50:56:87:86:b9 &g