本篇文章介绍如何规划及创建linuxbridge+flat网络,实现实例间及实例与外部的通讯。读者应对OpenStack及其网络服务Neutron有初步的了解。
1.规划网络
部署节点为一个controller节点(包含网络节点),两个compute节点。controller节点有3个网卡,分别为eth0(管理和API网络,CIDR为192.168.128.0/24)、eth1(租户网络,CIDR为10.10.10.0/24,不设置IP)、eth2(外部网络,CIDR为11.11.11.0/24,不设置IP);compute节点有2个网卡,分别为eth0(管理和API网络,CIDR为192.168.128.0/24)、eth1(租户网络,CIDR为10.10.10.0/24,不设置IP)。
这里用VMware Workstation下面的三个虚拟机分别作为controller、compute1、compute2节点,其网卡设置为:
- 每个主机的eth0网卡作为 API&Management 网络,使用的是“NAT模式” ,可以连接到外网下载OpenStack软件
- 每个主机的eth1网卡作为 Tenant(VM) 网络,实现VM间的通讯,使用的是“仅主机模式 ”
- controller节点也作为网络节点,需要模拟OpenStack的外网,增加网卡eth2,这里也选择“仅主机模式”
Neutron ML2的Type Driver为flat
Neutron ML2的Mechanism Driver为LinuxBridge
Neutron L2 Agent为LinuxBridge
2.创建虚拟网络
修改配置文件
/etc/neutron/plugins/ml2/ml2_conf.ini
controller节点 /etc/neutron/plugins/ml2/linuxbridge_agent.ini
注意:这里没有用到vxlan网络,enable_vxlan必须设置为False,否则会报错。
compute节点 /etc/neutron/plugins/ml2/linuxbridge_agent.ini
[securitygroup]和[vxlan]的配置同上
/etc/neutron/l3_agent.ini
/etc/neutron/dhcp_agent.ini
配置文件修改后,需要重启网络相关服务。
登陆到dashboard,可以查看网络服务信息。
controller节点启动了 neturon-metadata-agent、neutron-l3-agent、neutron-linuxbridge-agent、neutron-dhcp-agent
compute节点启动了 neutron-linuxbridge-agent
创建flat1网络,物理网络为provider(和前面的配置一致,否则报错),网络地址为10.10.10.0/24
创建外部网络,物理网络为external(和前面的配置一致,否则报错),网络地址为11.11.11.0/24
创建路由器,把flat网络和外部网络连通
我们的网络环境就准备好了。
3.测试网络
创建3个实例,都选择flat1网络。
其中:test1、test2被调度到compute1节点,test3被调度到compute2节点。
查看网络拓扑。flat1网络通过路由器连接到了外网。所以理论上test1、test2、test3能互通,并且也能连通外部网络。
为了保证外部能ping通以及能ssh登陆到实例,需要在安全组那加两条规则
下面测试网络的连通性:
同一flat网络:10.10.10.11 ping 10.10.10.16
外网: 10.10.10.11 ping 11.11.11.111
(确保外网相同网段有一台机器)
目前的网络结构如下
查看各个节点上面的虚拟网络设备
controller节点
root@controller:~# brctl show
bridge name bridge id STP enabled interfaces
brq30acf3ad-b9 8000.000c29c24683 no eth1
tap05de88b6-aa
tap28299cf0-82
brq662b5cb3-38 8000.000c29c2468d no eth2
tap88b95f4f-ea
virbr0 8000.52540057a3c4 yes virbr0-nic
compute1节点
root@compute1:~# brctl show
bridge name bridge id STP enabled interfaces
brq30acf3ad-b9 8000.000c29b8203b no eth1
tap77c04ce5-c1
tap806d032e-7b
virbr0 8000.52540066ffc3 yes virbr0-nic
compute2节点
root@compute2:~# brctl show
bridge name bridge id STP enabled interfaces
brq30acf3ad-b9 8000.000c29b969aa no eth1
tap8cc6c685-f9
virbr0 8000.52540066ffc3 yes virbr0-nic
controller节点1个Router、1个DHCP分别在各自的namespace下
root@controller:~# ip netns
qrouter-7a51a575-00f1-4709-badc-3d2866084c84
qdhcp-30acf3ad-b938-49cd-9fb1-81cb35559e23
可以通过exec查看router细节
root@controller:~# ip netns exec qrouter-7a51a575-00f1-4709-badc-3d2866084c84 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: qr-28299cf0-82@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:59:44:2f brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-28299cf0-82
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe59:442f/64 scope link
valid_lft forever preferred_lft forever
3: qg-88b95f4f-ea@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:0e:72:38 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.10/24 brd 11.11.11.255 scope global qg-88b95f4f-ea
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe0e:7238/64 scope link
valid_lft forever preferred_lft forever
root@controller:~# ip netns exec qrouter-7a51a575-00f1-4709-badc-3d2866084c84 route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 11.11.11.1 0.0.0.0 UG 0 0 0 qg-88b95f4f-ea
10.10.10.0 * 255.255.255.0 U 0 0 0 qr-28299cf0-82
11.11.11.0 * 255.255.255.0 U 0 0 0 qg-88b95f4f-ea
router的iptables
root@controller:~# ip netns exec qrouter-7a51a575-00f1-4709-badc-3d2866084c84 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i qg-88b95f4f-ea ! -o qg-88b95f4f-ea -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-88b95f4f-ea -j SNAT --to-source 11.11.11.10
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.10
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
可以通过exec查看dhcp细节
root@controller:~# ip netns exec qdhcp-30acf3ad-b938-49cd-9fb1-81cb35559e23 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ns-05de88b6-aa@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:77:12:d0 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.10/24 brd 10.10.10.255 scope global ns-05de88b6-aa
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-05de88b6-aa
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe77:12d0/64 scope link
valid_lft forever preferred_lft forever
再看下更详细的网络结构
controller节点
compute节点
如果需要从外部网络访问虚拟机,则要添加Floating IP
test1添加floating IP为11.11.11.11
测试网络连通:
外网:11.11.11.111 ping 11.11.11.11
root@ubuntu:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:0f brd ff:ff:ff:ff:ff:ff
inet 192.168.128.10/24 brd 192.168.128.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d0f/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:19 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.10/24 brd 192.168.1.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d19/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:0d:7d:23 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.111/24 brd 11.11.11.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe0d:7d23/64 scope link
valid_lft forever preferred_lft forever
root@ubuntu:~# ping 11.11.11.11
PING 11.11.11.11 (11.11.11.11) 56(84) bytes of data.
64 bytes from 11.11.11.11: icmp_seq=8 ttl=63 time=2.55 ms
64 bytes from 11.11.11.11: icmp_seq=9 ttl=63 time=2.32 ms
64 bytes from 11.11.11.11: icmp_seq=10 ttl=63 time=2.17 ms
64 bytes from 11.11.11.11: icmp_seq=11 ttl=63 time=1.95 ms
外网:11.11.11.111 ssh登录到 11.11.11.11
实际是登录到test1实例,IP为10.10.10.11
root@ubuntu:~# ssh cirros@11.11.11.11
The authenticity of host '11.11.11.11 (11.11.11.11)' can't be established.
RSA key fingerprint is 90:29:17:40:dc:b2:36:02:60:59:45:04:a6:38:c6:e3.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '11.11.11.11' (RSA) to the list of known hosts.
cirros@11.11.11.11's password:
$
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:1d:02:01 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.11/24 brd 10.10.10.255 scope global eth0
inet6 fe80::f816:3eff:fe1d:201/64 scope link
valid_lft forever preferred_lft forever
$ ping 10.10.10.16
PING 10.10.10.16 (10.10.10.16): 56 data bytes
64 bytes from 10.10.10.16: seq=0 ttl=64 time=10.765 ms
64 bytes from 10.10.10.16: seq=1 ttl=64 time=3.322 ms
64 bytes from 10.10.10.16: seq=2 ttl=64 time=2.035 ms
查看controller节点router的变化
root@controller:~# ip netns exec qrouter-7a51a575-00f1-4709-badc-3d2866084c84 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: qr-28299cf0-82@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:59:44:2f brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 brd 10.10.10.255 scope global qr-28299cf0-82
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe59:442f/64 scope link
valid_lft forever preferred_lft forever
3: qg-88b95f4f-ea@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:0e:72:38 brd ff:ff:ff:ff:ff:ff
inet 11.11.11.10/24 brd 11.11.11.255 scope global qg-88b95f4f-ea
valid_lft forever preferred_lft forever
inet 11.11.11.11/32 brd 11.11.11.11 scope global qg-88b95f4f-ea
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe0e:7238/64 scope link
valid_lft forever preferred_lft forever
iptables
root@controller:~# ip netns exec qrouter-7a51a575-00f1-4709-badc-3d2866084c84 iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-POSTROUTING
-N neutron-l3-agent-PREROUTING
-N neutron-l3-agent-float-snat
-N neutron-l3-agent-snat
-N neutron-postrouting-bottom
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 11.11.11.11/32 -j DNAT --to-destination 10.10.10.11
-A neutron-l3-agent-POSTROUTING ! -i qg-88b95f4f-ea ! -o qg-88b95f4f-ea -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-PREROUTING -d 11.11.11.11/32 -j DNAT --to-destination 10.10.10.11
-A neutron-l3-agent-float-snat -s 10.10.10.11/32 -j SNAT --to-source 11.11.11.11
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-88b95f4f-ea -j SNAT --to-source 11.11.11.10
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 11.11.11.10
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
这样我们就创建并且测试了LinuxBridge+flat网络,验证了网络的连通性。并对Neutron虚拟网络实现的细节进行了描述,希望对大家有帮助。
在Neutron的学习总结过程中,参考了网络上面的大量有价值的文档,在这里对无私分享的同学们表示衷心感谢!尤其是《每天5分钟学习OpenStack》,写的非常详细,本篇文档主要参考了里面的内容,在这里重点推荐:)。