Neutron 对虚拟三层网络的实现是通过其 L3 Agent (neutron-l3-agent)。该 Agent 利用 Linux IP 栈、route 和 iptables 来实现内网内不同网络内的虚机之间的网络流量,以及虚机和外网之间网络流量的路由和转发。为了在同一个Linux 系统上支持可能的 IP 地址空间重叠,它使用了 Linux network namespace 来提供隔离的转发上下文。
NameSpace技术
在二层网络上,VLAN 可以将一个物理交换机分割成几个独立的虚拟交换机。类似地,在三层网络上,Linux network namespace(netns) 可以将一个物理三层网络分割成几个独立的虚拟三层网络。
Network namespace (netns)从 Linux 2.6.24 版本开始添加,直到 2.6.29 添加完成。每个 netns 拥有独立的 (virtual)network devices, IP addresses, IP routing tables, /proc/net directory, ports 等等。新创建的 netns 默认只包含 loopback device。除了这个设备,每个 network device,不管是物理的还是虚拟的网卡还是网桥等,都只能存在于一个 netns。而且,连接物理硬件的物理设备只能存在于 root netns。其它普通的网络设备可以被创建和添加到某个 netns。
添加 network namespace
ip netnas add <network namespace name>
Example:
ip netns add nstest
列表所有 netns
ip netns list
删除某 netns
ip netns delete <network namespace name>
在 network namespace 中运行命令
ip netns exec <network namespace name> <command>
Example using the namespace from above:
ip netns exec nstest ip addr
添加 virtual interfaces 到 network namespace
ip link add veth-a type veth peer name veth-b #创建一对虚拟网卡veth-a 和 veth-b,两者由一根虚拟网线连接
将 veth-b 添加到 network namespace
ip link set veth-b netns nstest
设置 vi 的 IP 地址
ip netns exec nstest ip addr add 10.0.0.2/24 dev veth-b
ip netns exec nstest ip link set dev veth-b up
设置默认namespace的vi地址
ip addr add 10.0.0.1/24 dev veth-a
ip link set dev veth-a up
ping
ip netns exec nstest ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.054 ms
查看路由表和 iptbales
ip netns exec nstest route
ip netns exec nstest iptables -L
Iptables
Neutron 主要用到 filter 表和 nat 表,其中, filter 用来实现安全组(Security Group)和 防火墙(FWaas);nat 主要用来实现 router。
iptables其实是个client,供用户去管理防火墙。相关的请求最后会发送相关内核模块,如ip_tables。ip_tables内核模块主要用于组织iptables使用的表,链,规则。netfilter是一套技术框架,ip_tables依托于netfilter来注册各种hooks实现对数据包的具体控制。一些厂商的防火墙,入侵检测,入侵防御系统什么的基本依托于Netfilter来实现(从事过相关开发)。
NEW,ESTABLISHED,RELATED,INVALID状态
NEW: conntrack模块看到的某个连接第一个包,它即将被匹配了。比如,我们看到一个SYN包,是我们所留意的连接的第一个包,就要匹配它。第一个包也可能不是SYN包,但它仍会被认为是NEW状态。
ESTABLISHED: 已经注意到两个方向上的数据传输,而且会继续匹配这个连接的包。处于ESTABLISHED状态的连接是非常容易理解的。只要发送并接到应答,连接就是ESTABLISHED的了。一个连接要从NEW变为ESTABLISHED,只需要接到应答包即可,不管这个包是发往防火墙的,还是要由防火墙转发的。ICMP的错误和重定向等信息包也被看作是ESTABLISHED,只要它们是我们所发出的信息的应答。
RELATED 当一个连接和某个已处于ESTABLISHED状态的连接有关系时,就被认为是RELATED的了。换句话说,一个连接要想是RELATED的,首先要有一个ESTABLISHED的连接。这个ESTABLISHED连接再产生一个主连接之外的连接,这个新的连接就是RELATED的了,比如ftp的父子链接。
INVALID 非以上状态的包。
iptables自定义连
如果想自行定义规则链,可以通过-N参数,然后通过-j/–jump跳转过来就可以了。
iptables -N 链名
iptables ... -j 链名
下面我们查看openstack中的router是如何使用iptables的。
首先当路由器创建完成并加入网络(9.9.9.0/24)之后,我们查看一下它iptable信息。
# Generated by iptables-save v1.6.0 on Tue Jul 18 21:49:49 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Tue Jul 18 21:49:49 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 21:49:49 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-b111bb2e-84 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
COMMIT
# Completed on Tue Jul 18 21:49:49 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 21:49:49 2017
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Tue Jul 18 21:49:49 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 21:49:49 2017
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-scope
-A neutron-l3-agent-PREROUTING -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-floatingip
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-A neutron-l3-agent-float-snat -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-scope -i qr-b111bb2e-84 -j MARK --set-xmark 0x4000000/0xffff0000
COMMIT
# Completed on Tue Jul 18 21:49:49 2017
输出是分段的,每个段落一个表,*开头后面跟着表名。
:开头的行是对链匹配次数的总结,后面跟着统计信息,[数据包:字节数]。
后面跟着的是具体规则。
最后跟着COMMIT表示一个表的结束。
然后我们将该路由器关联一个外部网络(10.0.99.0/24),我们发现该网络的所有虚拟机访问外网时候,会将凡是到达外网络的数据都进行SNAT为其外部地址10.0.99.2,但是没有相应的表项导致外网中的虚拟机是无法访问内部虚拟机的。
root@netagent:~# ip netns exec qrouter-71d8932f-3782-470f-b3a2-f80203f96885 iptables-save
# Generated by iptables-save v1.6.0 on Tue Jul 18 22:14:04 2017
*nat
:PREROUTING ACCEPT [98:22227]
:INPUT ACCEPT [1:325]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-POSTROUTING ! -i qg-37790cfc-43 ! -o qg-37790cfc-43 -m conntrack ! --ctstate DNAT -j ACCEPT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat
-A neutron-l3-agent-snat -o qg-37790cfc-43 -j SNAT --to-source 10.0.99.2
-A neutron-l3-agent-snat -m mark ! --mark 0x2/0xffff -m conntrack --ctstate DNAT -j SNAT --to-source 10.0.99.2
-A neutron-postrouting-bottom -m comment --comment "Perform source NAT on outgoing traffic." -j neutron-l3-agent-snat
COMMIT
# Completed on Tue Jul 18 22:14:04 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 22:14:04 2017
*filter
:INPUT ACCEPT [26:8892]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
:neutron-l3-agent-scope - [0:0]
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope
-A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT
-A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP
-A neutron-l3-agent-scope -o qr-b111bb2e-84 -m mark ! --mark 0x4000000/0xffff0000 -j DROP
COMMIT
# Completed on Tue Jul 18 22:14:04 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 22:14:04 2017
*raw
:PREROUTING ACCEPT [123:30794]
:OUTPUT ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
COMMIT
# Completed on Tue Jul 18 22:14:04 2017
# Generated by iptables-save v1.6.0 on Tue Jul 18 22:14:04 2017
*mangle
:PREROUTING ACCEPT [118:29354]
:INPUT ACCEPT [23:7908]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-floatingip - [0:0]
:neutron-l3-agent-mark - [0:0]
:neutron-l3-agent-scope - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A INPUT -j neutron-l3-agent-INPUT
-A FORWARD -j neutron-l3-agent-FORWARD
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A neutron-l3-agent-POSTROUTING -o qg-37790cfc-43 -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-mark
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-scope
-A neutron-l3-agent-PREROUTING -m connmark ! --mark 0x0/0xffff0000 -j CONNMARK --restore-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-PREROUTING -j neutron-l3-agent-floatingip
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x1/0xffff
-A neutron-l3-agent-float-snat -m connmark --mark 0x0/0xffff0000 -j CONNMARK --save-mark --nfmask 0xffff0000 --ctmask 0xffff0000
-A neutron-l3-agent-mark -i qg-37790cfc-43 -j MARK --set-xmark 0x2/0xffff
-A neutron-l3-agent-scope -i qg-37790cfc-43 -j MARK --set-xmark 0x4000000/0xffff0000
-A neutron-l3-agent-scope -i qr-b111bb2e-84 -j MARK --set-xmark 0x4000000/0xffff0000
COMMIT
# Completed on Tue Jul 18 22:14:04 2017
Floating Ip
我们将9.9.9.0/24网络中的虚拟机(9.9.9.3)分配一个floatingIP(10.0.99.3),查看router的iptables-save。
root@netagent:~# ip netns exec qrouter-71d8932f-3782-470f-b3a2-f80203f96885 iptables-save
# Generated by iptables-save v1.6.0 on Tue Jul 18 22:23:42 2017
*nat
:PREROUTING ACCEPT [48:12000]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-POSTROUTING - [0:0]
:neutron-l3-agent-PREROUTING - [0:0]
:neutron-l3-agent-float-snat - [0:0]
:neutron-l3-agent-snat - [0:0]
:neutron-postrouting-bottom - [0:0]
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING
-A POSTROUTING -j neutron-postrouting-bottom
-A neutron-l3-agent-OUTPUT -d 10.0.99.3/32 -j DNAT --to-destination 9.9.9.3
-A neutron-l3-agent-POSTROUTING ! -i qg-37790cfc-43 ! -o qg-37790cfc-43 -m connt