The OpenStack Networking components are deployed on the Controller, Compute, and Network nodes in the following configuration:
In case of Two Node Development Cluster :-
Controller node: hosts the Neutron server service, which provides the networking API and communicates with and tracks the agents.
DHCP agent: spawns and controls dnsmasq processes to provide leases to instances. This agent also spawns neutron-ns-metadata-proxy processes as part of the metadata system.
Metadata agent: Provides a metadata proxy to the nova-api-metadata service. The neutron-ns-metadata-proxy direct traffic that they receive in their namespaces to the proxy.
OVS plugin agent: Controls OVS network bridges and routes between them via patch, tunnel, or tap without requiring an external OpenFlow controller.
L3 agent: performs L3 forwarding and NAT.
In case of Three Node or more ( several Compute Nodes) :-
Separate box hosts Neutron Server and all services mentioned above
Compute node: has an OVS plugin agent and openstack-nova-compute service.
Namespaces (View Identifying and Troubleshooting Neutron Namespaces )
For each network you create, the Network node (or Controller node, if combined) will have a unique network namespace (netns) created by the DHCP and Metadata agents. The netnshosts an interface and IP addresses for dnsmasq and the neutron-ns-metadata-proxy. You can view the namespaces with the `ip netns list` command, and can interact with the namespaces with the `ip netns exec
namespace
command
` command.Every l2-agent/private network has an associated dhcp namespace and
Every l3-agent/router has an associated router namespace.
Network namespace starts with dhcp- followed by the ID of the network.
Router namespace starts with qrouter- followed by the ID of the router.
Source admin credentials and get network list
[root@dfw02 ~(keystone_admin)]$ neutron net-list
+--------------------------------------+------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+------+-----------------------------------------------------+
| 1eea88bb-4952-4aa4-9148-18b61c22d5b7 | int | fa930cea-3d51-4cbe-a305-579f12aa53c0 10.0.0.0/24 |
| 426bb226-0ab9-440d-ba14-05634a17fb2b | int1 | 9e0d457b-c4c4-45cf-84e2-4ac7550f3b06 40.0.0.0/24 |
| 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | ext | f30e5a16-a055-4388-a6ea-91ee142efc3d 192.168.1.0/24 |
+--------------------------------------+------+-----------------------------------------------------+
Using command `ip netns list` run following commands to get tenants
qdhcp-* names
[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 1eea88bb-4952-4aa4-9148-18b61c22d5b7
qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7
[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 426bb226-0ab9-440d-ba14-05634a17fb2b
qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b
Check tenants Namespace via getting IP and ping this IP inside namespaces
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ifconfig
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 0 (Local Loopback)
RX packets 35 bytes 4416 (4.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 35 bytes 4416 (4.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ns-343b0090-24: flags=4163 mtu 1500
inet 40.0.0.3 netmask 255.255.255.0 broadcast 40.0.0.255
inet6 fe80::f816:3eff:fe01:8b55 prefixlen 64 scopeid 0x20
ether fa:16:3e:01:8b:55 txqueuelen 1000 (Ethernet)
RX packets 3251 bytes 386284 (377.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1774 bytes 344082 (336.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ping -c 3 40.0.0.3
PING 40.0.0.3 (40.0.0.3) 56(84) bytes of data.
64 bytes from 40.0.0.3: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 40.0.0.3: icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from 40.0.0.3: icmp_seq=3 ttl=64 time=0.032 ms
--- 40.0.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.032/0.040/0.047/0.006 ms
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 40.0.0.3:53 0.0.0.0:* LISTEN
tcp6 0 0 fe80::f816:3eff:fe01:53 :::* LISTEN
udp 0 0 40.0.0.3:53 0.0.0.0:*
udp 0 0 0.0.0.0:67 0.0.0.0:*
udp6 0 0 fe80::f816:3eff:fe01:53 :::*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
Now verify that we have a copy of dnsmasq process to support every tenants namespace
[root@dfw02 ~(keystone_admin)]$ ps -aux | grep dhcp
neutron 2320 0.3 0.3 263908 30696 ? Ss 08:18 2:14 /usr/bin/python /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --log-file /var/log/neutron/dhcp-agent.log
nobody 3529 0.0 0.0 15532 832 ? S 08:20 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-40dd712c-e4 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/host --dhcp-optsfile=/var/lib/neutron/dhcp/1eea88bb-4952-4aa4-9148-18b61c22d5b7/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
nobody 3530 0.0 0.0 15532 944 ? S 08:20 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-343b0090-24 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/host --dhcp-optsfile=/var/lib/neutron/dhcp/426bb226-0ab9-440d-ba14-05634a17fb2b/opts --leasefile-ro --dhcp-range=set:tag0,40.0.0.0,static,120s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal
List interfaces inside dhcp namespace
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ns-343b0090-24: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether fa:16:3e:01:8b:55 brd ff:ff:ff:ff:ff:ff
inet 40.0.0.3/24 brd 40.0.0.255 scope global ns-343b0090-24
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe01:8b55/64 scope link
valid_lft forever preferred_lft forever
(A) From the instance to a router
Check routing inside dhcp namespace
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ip r
default via 40.0.0.1 dev ns-343b0090-24
40.0.0.0/24 dev ns-343b0090-24 proto kernel scope link src 40.0.0.3
Check routing inside the router namespace
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 ip r
default via 192.168.1.1 dev qg-9c090153-08
40.0.0.0/24 dev qr-e031db6b-d0 proto kernel scope link src 40.0.0.1
192.168.1.0/24 dev qg-9c090153-08 proto kernel scope link src 192.168.1.114
Get routers list via similar grep and network-id to obtain Routers Namespaces
[root@dfw02 ~(keystone_admin)]$ neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 86b3008c-297f-4301-9bdc-766b839785f1 | router2 | {"network_id": "780ce2f3-2e6e-4881-bbac-857813f9a8e0", "enable_snat": true} |
| bf360d81-79fb-4636-8241-0a843f228fc8 | router1 | {"network_id": "780ce2f3-2e6e-4881-bbac-857813f9a8e0", "enable_snat": true} |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
Now get qrouter-* namespaces via `ip netns list` command :-
[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 86b3008c-297f-4301-9bdc-766b839785f1
qrouter-86b3008c-297f-4301-9bdc-766b839785f1
[root@dfw02 ~(keystone_admin)]$ ip netns list | grep bf360d81-79fb-4636-8241-0a843f228fc8
qrouter-bf360d81-79fb-4636-8241-0a843f228fc8
Now verify L3 forwarding & NAT via command `iptables -L -t nat` inside router namespace and check routing port 80 for 169.254.169.254 to the RDO Havana Controller's ( in my configuration running Neutron Server Service along with all agents) host at metadata port
8700
(B) ( through a NAT rule in the router namespace)
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-PREROUTING all -- anywhere anywhere
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-OUTPUT all -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-POSTROUTING all -- anywhere anywhere
neutron-postrouting-bottom all -- anywhere anywhere
Chain neutron-l3-agent-OUTPUT (1 references)
target prot opt source destination
DNAT all -- anywhere dfw02.localdomain to:40.0.0.2
DNAT all -- anywhere dfw02.localdomain to:40.0.0.6
DNAT all -- anywhere dfw02.localdomain to:40.0.0.5
Chain neutron-l3-agent-POSTROUTING (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere ! ctstate DNAT
Chain neutron-l3-agent-PREROUTING (1 references)
target prot opt source destination
REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 8700
DNAT all -- anywhere dfw02.localdomain to:40.0.0.2
DNAT all -- anywhere dfw02.localdomain to:40.0.0.6
DNAT all -- anywhere dfw02.localdomain to:40.0.0.5
Chain neutron-l3-agent-float-snat (1 references)
target prot opt source destination
SNAT all -- 40.0.0.2 anywhere to:192.168.1.107
SNAT all -- 40.0.0.6 anywhere to:192.168.1.104
SNAT all -- 40.0.0.5 anywhere to:192.168.1.110
Chain neutron-l3-agent-snat (1 references)
target prot opt source destination
neutron-l3-agent-float-snat all -- anywhere anywhere
SNAT all -- 40.0.0.0/24 anywhere to:192.168.1.114
Chain neutron-postrouting-bottom (1 references)
target prot opt source destination
neutron-l3-agent-snat all -- anywhere anywhere
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-PREROUTING all -- anywhere anywhere
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-OUTPUT all -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
neutron-l3-agent-POSTROUTING all -- anywhere anywhere
neutron-postrouting-bottom all -- anywhere anywhere
Chain neutron-l3-agent-OUTPUT (1 references)
target prot opt source destination
DNAT all -- anywhere dfw02.localdomain to:10.0.0.2
Chain neutron-l3-agent-POSTROUTING (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere ! ctstate DNAT
Chain neutron-l3-agent-PREROUTING (1 references)
target prot opt source destination
REDIRECT tcp -- anywhere 169.254.169.254 tcp dpt:http redir ports 8700
DNAT all -- anywhere dfw02.localdomain to:10.0.0.2
Chain neutron-l3-agent-float-snat (1 references)
target prot opt source destination
SNAT all -- 10.0.0.2 anywhere to:192.168.1.103
Chain neutron-l3-agent-snat (1 references)
target prot opt source destination
neutron-l3-agent-float-snat all -- anywhere anywhere
SNAT all -- 10.0.0.0/24 anywhere to:192.168.1.100
Chain neutron-postrouting-bottom (1 references)
target prot opt source destination
neutron-l3-agent-snat all -- anywhere anywhere
Verifying service listening at 8700 port inside routers namespaces
output seems like this :-
(C) (to an instance of the neutron-ns-metadata-proxy)
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 netstat -lntp | grep 8700
tcp 0 0 0.0.0.0:8700 0.0.0.0:* LISTEN 4946/python
Check process with pid 4946
[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 4946
root 4946 1 0 08:58 ? 00:00:00 /usr/bin/python /bin/neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/86b3008c-297f-4301-9bdc-766b839785f1.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=86b3008c-297f-4301-9bdc-766b839785f1 --state_path=/var/lib/neutron --metadata_port=8700 --verbose --log-file=neutron-ns-metadata-proxy-86b3008c-297f-4301-9bdc-766b839785f1.log --log-dir=/var/log/neutron
root 10396 11489 0 16:33 pts/3 00:00:00 grep --color=auto 4946
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 netstat -lntp | grep 8700
tcp 0 0 0.0.0.0:8700 0.0.0.0:* LISTEN 4746/python
Check process with pid 4746
[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 4746
root 4746 1 0 08:58 ? 00:00:00 /usr/bin/python /bin/neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/bf360d81-79fb-4636-8241-0a843f228fc8.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=bf360d81-79fb-4636-8241-0a843f228fc8 --state_path=/var/lib/neutron --metadata_port=8700 --verbose --log-file=neutron-ns-metadata-proxy-bf360d81-79fb-4636-8241-0a843f228fc8.log --log-dir=/var/log/neutron
Now run following commands inside routers namespaces to check status of neutron-metadata port :-
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:8700 0.0.0.0:* LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:8700 0.0.0.0:* LISTEN
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
Outside routers namespace it would look like
(D) (to the actual Nova metadata service)
Run on Controller , hosting Neutron Server Service :-
[root@dfw02 ~(keystone_admin)]$ netstat -lntp | grep 8700
tcp 0 0 0.0.0.0:8700 0.0.0.0:* LISTEN 2746/python
Check process with pid 2746
[root@dfw02 ~(keystone_admin)]$ ps -ef | grep 2746
nova 2746 1 0 08:57 ? 00:02:31 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova 2830 2746 0 08:57 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova 2851 2746 0 08:57 ? 00:00:10 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
nova 2858 2746 0 08:57 ? 00:00:02 /usr/bin/python /usr/bin/nova-api --logfile /var/log/nova/api.log
root 9976 11489 0 16:31 pts/3 00:00:00 grep --color=auto 2746
So , we actually verified statement from Direct_access _to_Nova_metadata
in an environment running Neutron, a request from your instance must traverse a number of steps:
1. From the instance to a router,(A)
2. Through a NAT rule in the router namespace, (B)
3. To an instance of the neutron-ns-metadata-proxy, (C)
4. To the actual Nova metadata service (D)
References
1. OpenStack Networking concepts