oVirt构筑openstack vm无法访问外网的解决办法

用两台物理机构筑RHEV。然后创建3台vm用于部署osp

vm1：controller

vm2：nova1

vm3：nova2

 

部署完成后创建租户网络：tenant1和tenant2。

创建外网：ext-net

创建路由：router

tenant1和tenant2加入router，并且吧ext-net设置为router网关。

场景如下：

1. tenant1和tenant2可以互通

2. tenant1/tenant2可以ping通router内的gateway设备地址

3. tenant1/tenant2可以ping通controller（vm1）的网卡地址

4. tenant1/tenant2可以ping不通nova1/2（vm2/3）的网卡地址

 

调查后发现，router的namespace里面只能ping通controller：

[root@controller ~]# ip netns exec $qrouter ping 10.0.0.51
PING 10.0.0.51 (10.0.0.51) 56(84) bytes of data.
64 bytes from 10.0.0.51: icmp_seq=1 ttl=64 time=0.504 ms
64 bytes from 10.0.0.51: icmp_seq=2 ttl=64 time=0.066 ms

[root@controller ~]# ip netns exec $qrouter ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
From 10.0.0.101 icmp_seq=1 Destination Host Unreachable
From 10.0.0.101 icmp_seq=2 Destination Host Unreachable
From 10.0.0.101 icmp_seq=3 Destination Host Unreachable

继续调查，发现controller的网卡可以接收到router发来的数据包

[root@controller ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
02:30:45.498264 ARP, Request who-has 10.0.0.101 tell 10.0.0.51, length 28
02:44:07.638045 ARP, Request who-has 10.0.0.21 tell 10.0.0.41, length 46
02:47:49.901267 ARP, Request who-has 10.0.0.1 tell 10.0.0.101, length 28
02:47:50.903910 ARP, Request who-has 10.0.0.1 tell 10.0.0.101, length 28

 

但是其他机器收不到router的包：

[root@nova1 ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
照理说这是广播的，不应该收不到。所以判断是controller没有继续转发router的arp包。

 

检查ip forward，发现已经启用。

net.ipv4.ip_forward = 1
继续调查可能问题出在RHEV的设置。

参考 http://lists.ovirt.org/pipermail/users/2013-July/015259.html 之后，进行如下操作，故障排除：

1. RHEV hypervisor安装macproof

yum install -y vdsm-hook-promisc vdsm-hook-nestedvt vdsm-hook-macspoof

 

2. RHEVM 启用macproof

[root@centos80 ~]# engine-config -s "UserDefinedVMProperties=macspoof=(true|false)"
Please select a version:
1. 3.0
2. 3.1
3. 3.2
4. 3.3
5. 3.4
6. 3.5
7. 3.6
7
[root@centos80 ~]# systemctl restart ovirt-engine

3. Admin Portal界面修改controller 虚拟机Custome Properties

   设置 macspoof = true

 

4. 登录controller确认网络

[root@controller ~]# ip netns exec $qrouter ping 10.0.0.41
PING 10.0.0.41 (10.0.0.41) 56(84) bytes of data.
64 bytes from 10.0.0.41: icmp_seq=1 ttl=64 time=1.12 ms
64 bytes from 10.0.0.41: icmp_seq=2 ttl=64 time=0.557 ms

原因分析：
（http://iranzo.github.io/blog/2015/07/17/rhev-m-with-nested-vm-for-osp/）

RHEV also includes a security feature that makes it impossible for a VM to spy on the communications meant to other VM’s that makes it impossible to simulate other MAC’s within it, and this is performed via libvirt filters on the interfaces.

To come to our rescue, another hook comes to play in, this time macspoof which allows to disable this security measure for a VM so it can execute virtualization within.