keepalive 在openstack nova kvm中的应用

在一些项目中，会遇到到HA相关的问题，而keepalive是一个应用广泛的功能模块，随着openstack在云计算中的普通，越来越多的项目迁移进了openstack创建的kvm环境中。

声明 : 此文档只做学习交流使用,请勿用作其他商业用途
author : 朝阳_tony
E-mail : linzhaolover@163.com
Create Date: 2016-2-27 14:10:01
Last Change: 2016-2-27 14:50:08
转载请注明出处：http://blog.csdn.net/linzhaolover

创建kvm

用nova 在openstack 环境中创建3个kvm，baseimage: RHEL 7.1 ppc64le ，都要挂载internal ip 和 floating ip

nova list
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
| ID                                   | Name                              | Status | Task State | Power State | Networks                             |
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
| ef1f1a66-1375-4d9a-b99b-580c8340b59d | linzhbj-kvm.ppc64le-keepalived-n0 | ACTIVE | -          | Running     | ent_vlan=192.168.33.14, 172.16.0.237 |
| 3eb18869-97c5-42cb-94e7-466d52d4199d | linzhbj-kvm.ppc64le-keepalived-n1 | ACTIVE | -          | Running     | ent_vlan=192.168.33.15, 172.16.0.238 |
| daf4b686-c33d-48e5-89cc-0a0f99d8fb73 | linzhbj-kvm.ppc64le-keepalived-n2 | ACTIVE | -          | Running     | ent_vlan=192.168.33.91, 172.16.0.239 |
+--------------------------------------+-----------------------------------+--------+------------+-------------+--------------------------------------+
分别创建internal ip 和 floating ip 一会用做总的virtual ip

neutron port-create internal_vlan
nova  floating-ip-create floating_vlan

neutron port-list
| 5c09376e-bd65-4aed-9f4e-dd6ba2879a27 | linzhbj_keepalived_vip0      | fa:16:3e:4f:47:21 | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.91"} |
| 0e0c48e6-2e32-48c5-a01d-c059c4931064 | linzhbj_keepalived_vip1      | fa:16:3e:f9:4f:41 | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.14"} |
| b133aa34-a2fc-4c05-a962-18118de6db83 | linzhbj_keepalived_vip2      | fa:16:3e:aa:02:ea | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.15"} |
| 5ffd38c7-9ec3-4661-84f9-048eda70f738 | linzhbj_keepalived_vip_total | fa:16:3e:ea:81:6a | {"subnet_id": "585b7e7e-e890-4b4b-90f9-e877605f5d5e", "ip_address": "192.168.33.92"} |
nova floating-ip-list
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
| Id                                   | IP           | Server Id                            | Fixed IP      | Pool    |
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
| 0730073b-9758-409f-843a-0f40e8f9b300 | 172.16.0.238 | 3eb18869-97c5-42cb-94e7-466d52d4199d | 192.168.33.15 | ext_net |
| 1c9c3745-d7d4-435c-b8c5-c19d415fde71 | 172.16.0.239 | daf4b686-c33d-48e5-89cc-0a0f99d8fb73 | 192.168.33.91 | ext_net |
| 7d056ad6-b2d9-47ae-92f3-d383d2aab366 | 172.16.1.60  |                                      | 192.168.33.92 | ext_net |
| a948ef38-01b6-4922-8025-067c0c610f17 | 172.16.0.237 | ef1f1a66-1375-4d9a-b99b-580c8340b59d | 192.168.33.14 | ext_net |
+--------------------------------------+--------------+--------------------------------------+---------------+---------+
将floating ip 与internal ip进行挂载，
neutron floatingip-associate  7d056ad6-b2d9-47ae-92f3-d383d2aab366 5ffd38c7-9ec3-4661-84f9-048eda70f738

配置keepalived

进入每个kvm安装keepalived，并进行配置启动

yum install keepalived

编辑keepalived 配置

vim /etc/keepalived/keepalived.conf
global_defs
{
    notification_email
    {
        linzhaolover@163.com
    }
    notification_email_from linzhaolover@163.com
    smtp_server 127.0.0.1
    stmp_connect_timeout 30
    router_id NodeA
}

vrrp_instance lnmp {
            state BACKUP
    interface eth0
    virtual_router_id 100
    priority 170
    advert_int 5
    track_interface {
        eth0
    }
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
      192.168.33.92/20 dev eth0
    }
}

service keepalived restart
ip addr

[root@host-192-168-33-15 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1454 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:aa:02:ea brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.15/20 brd 192.168.47.255 scope global dynamic eth0
       valid_lft 85999sec preferred_lft 85999sec
    inet 192.168.33.92/20 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feaa:2ea/64 scope link
       valid_lft forever preferred_lft forever

查看keepalive 产生的log
tail -f /var/log/messages

三个kvm都进行上面keepalived配置后，测试通不通，
ping 192.168.33.92
居然不通，看来还缺少东西，难道少iptable 规则

重点 config allow pairs

#   neutron port-update  <port-uuid> --allowed-address-pairs type=dict list=true mac_address=<mac_address>,ip_address=<ip_cidr>
mac_address 是kvm的mac地址，
ip_address 是 virtual ip 的地址，这一定不要搞错了

neutron port-update  5c09376e-bd65-4aed-9f4e-dd6ba2879a27 --name linzhbj_keepalived_vip0 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:4f:47:21",ip_address=192.168.33.92
neutron port-update  0e0c48e6-2e32-48c5-a01d-c059c4931064 --name linzhbj_keepalived_vip1 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:f9:4f:41",ip_address=192.168.33.92
neutron port-update  b133aa34-a2fc-4c05-a962-18118de6db83 --name linzhbj_keepalived_vip2 --allowed-address-pairs type=dict list=true mac_address="fa:16:3e:aa:02:ea",ip_address=192.168.33.92

在compute node 中check iptables规则
iptables -S | grep 192.168.33.92
-A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-s5c09376e-b -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-sb133aa34-a -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:AA:02:EA -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
这是对应每个kvm的mac的 iptables rule

总结

openstack中的kvm，要想正常运行，跑相应的服务，必须在 neutron 中配置相应的规则才能够允许运行。虽然很麻烦，但起到了很好的保护作用；

动动脑筋，办法总是比问题多，O(∩_∩)O~

bug

port-update 的时候会遇到bug

NT-628A69C Policy doesn't allow (rule:update_port and rule:update_port:allowed_address_pairs) to be performed.

vim /etc/neutron/policy.json
change 
"update_port:allowed_address_pairs": "rule:admin_or_network_owner",
to 
"update_port:allowed_address_pairs": "rule:admin_or_owner",

无论nova 还是neutron 都有自己的policy，所以你要设定好才可以

系统没有各个kvm iptables 规则

vim /etc/sysctl.conf
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0

使配置生效sysctl -p

vim /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

重启agent服务
service  neutron-plugin-openvswitch-agent restart

# iptables -S | grep 192.168.33
-A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-s0e0c48e6-2 -s 192.168.33.14/32 -m mac --mac-source FA:16:3E:F9:4F:41 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-s5c09376e-b -s 192.168.33.92/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN
-A neutron-openvswi-s5c09376e-b -s 192.168.33.91/32 -m mac --mac-source FA:16:3E:4F:47:21 -m comment --comment "Allow traffic from defined IP/MAC pairs." -j RETURN