openstack+opendaylight+ovsdb部署

环境：centos7+openstack-kilo+opendaylight-distribution-karaf-0.3.1-Lithium-SR1+ovsdb管理网络


controlnode:192.168.88.162
nova1:192.168.88.166
nova2:192.168.88.167
networknode:192.168.88.168
sdn controller:192.168.88.162（和openstack controller是一个）


租户网络类型是vxlan




参考官方文档：openstack_20150629.pdf/bk_getting_started_guide_20150629.pdf
其它：OpenDaylight OVSDB Plugin Network_OpenDaylight_Forum-Bangalore-Apr-2015-Ovsdb.pdf/The OpenDaylight OVSDB Project-July-2015_shague v1.pdf




SDN控制节点需要java环境:
步骤1
Prerequisites: OpenDaylight requires Java 1.7.0.
[root@controller odl_tools]# java -version
java version "1.7.0_85"


注意1：事先照openstack官方文档安装好openstack环境，这里不表述；


On the control host, Download the latest OpenDaylight release (at the time of writing,this is 0.2.1-Helium-SR1.1）
注意2：文档上是0.2.1，其实当前环境的SDN版本是最新的distribution-karaf-0.3.1-Lithium-SR1（2015.10.10)


Uncompress it as root, and start OpenDaylight (you can start OpenDaylight by runningkaraf directly, but exiting from the shell will shut it down):
$ tar xvfz distribution-karaf-0.2.1-Helium-SR1.1.tar.gz
$ cd distribution-karaf-0.2.0-Helium
$ ./bin/start                         # Start OpenDaylight as a server process
Connect to the Karaf shell, and install the odl-ovsdb-openstack bundle, dlux and theirdependencies:
$ ./bin/client                        # Connect to OpenDaylight with the client
opendaylight-user@root> feature:install odl-base-all odl-aaa-authn odl-restconf odl-nsf-all odl-adsal-northbound odl-mdsal-apidocs odl-ovsdb-openstack odl-ovsdb-northbound odl-dlux-core


注意3：直接使用bin/client需要输入密码，目前不确定密码是多少；可以直接执行bin/karaf启用ODL，直接登录karaf控制台，缺点是终端断的话，ODL就挂了；
注意4：直接安装上面的feature，br-int创建不出来，可能是模块间有冲突；feature:install odl-ovsdb-openstack  odl-ovsdb-northbound仅安装这两个目前没发现问题，缺点是没有页面显示；
       如果安装完上面的模块后发现不好使，uninstall多余的模块好像也不好使，此时可以将整个ODL目录删掉，重新解压即可；
opendaylight-user@root>feature:list -i|grep ovsdb
odl-ovsdb-library                    | 1.1.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OVSDB :: Library                                  
odl-ovsdb-schema-openvswitch         | 1.1.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OVSDB :: Schema :: Open_vSwitch                   
odl-ovsdb-schema-hardwarevtep        | 1.1.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OVSDB :: Schema :: hardware_vtep                  
odl-ovsdb-plugin                     | 1.1.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OpenDaylight :: OVSDB :: Plugin                   
odl-ovsdb-northbound                 | 0.7.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OpenDaylight :: OVSDB :: Northbound               
odl-ovsdb-openstack                  | 1.1.1-Lithium-SR1   | x         | ovsdb-1.1.1-Lithium-SR1                    | OpenDaylight :: OVSDB :: OpenStack Network Virtual
odl-ovsdb-southbound-api             | 1.1.1-Lithium-SR1   | x         | odl-ovsdb-southbound-1.1.1-Lithium-SR1     | OpenDaylight :: southbound :: api                 
odl-ovsdb-southbound-impl            | 1.1.1-Lithium-SR1   | x         | odl-ovsdb-southbound-1.1.1-Lithium-SR1     | OpenDaylight :: southbound :: impl                
odl-ovsdb-southbound-impl-rest       | 1.1.1-Lithium-SR1   | x         | odl-ovsdb-southbound-1.1.1-Lithium-SR1     | OpenDaylight :: southbound :: impl :: REST        
odl-ovsdb-southbound-impl-ui         | 1.1.1-Lithium-SR1   | x         | odl-ovsdb-southbound-1.1.1-Lithium-SR1     | OpenDaylight :: southbound :: impl :: UI


opendaylight-user@root>bundle:list -s|grep ovsdb
255 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.southbound-api                                    
256 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.southbound-impl                                   
283 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.utils.servicehelper                               
284 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.openstack.net-virt                                
285 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.openstack.net-virt-providers                      
350 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.library                                           
351 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.schema.openvswitch                                
352 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.schema.hardwarevtep                               
353 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.plugin                                            
354 | Active   |  80 | 1.1.1.Lithium-SR1                         | org.opendaylight.ovsdb.plugin-shell                                      
399 | Active   |  80 | 0.7.1.Lithium-SR1                         | org.opendaylight.ovsdb.northbound


If everything is installed correctly, you should now be able to log in to the dlux interfaceon http://$CONTROL_HOST:8181/dlux/index.html - the default username and password is "admin/admin" 
注意5：这一步由于没有安装dlux，会登录失败，不过不影响虚拟网络的创建和虚拟机之间的通信；






步骤2：清除openstack环境中的neutron配置


注意6：由于用SDN控制器来管理openstack网络，计算节点的ovs-agent就不需要了，neutron相关的api都是通过neutron-server--〉networking_odl-->neutron/plugins/ml2/drivers/opendaylight/driver.py
将请求转发给ODL来处理，ODL有多种虚拟网络管理方式（OVSDB/VTN等）；
所以需要在neutron-server所在的节点（一般是控制节点）安装pip install networking-odl，这个在文档中没有看到；
下面涉及network/router/实例的操作可以在openstack的管理页面执行；


Ensuring OpenStack network state is clean
When using OpenDaylight as the Neutron back-end, ODL expects to be the only source of truth for Open vSwitch configuration. Because of this, it is necessary to remove existing
OpenStack and Open vSwitch configurations to give OpenDaylight a clean slate.
Delete instances
$ nova list
$ nova delete <instance names> 


Remove link from subnets to routers
$ neutron subnet-list
$ neutron router-list
$ neutron router-port-list <router name>
$ neutron router-interface-delete <router name> <subnet ID or name>


Delete subnets, nets, routers
$ neutron subnet-delete <subnet name>
$ neutron net-list
$ neutron net-delete <net name>
$ neutron router-delete <router name>


Check that all ports have been cleared - at this point, this should be an empty list
$ neutron port-list




Ensure Neutron is stopped
While Neutron is managing the OVS instances on compute and control nodes,
OpenDaylight and Neutron can be in conflict. To prevent issues, we turn off Neutron server
on the network controller, and Neutron’s Open vSwitch agents on all hosts.




暂时停掉neutron-server服务
Turn off neutron-server on control node
# systemctl stop neutron-server


On each node in the cluster, shut down and disable Neutron’s agent services to ensure
that they do not restart after a reboot:
# systemctl stop neutron-openvswitch-agent
# systemctl disable neutron-openvswitch-agent


清除原有的ovs数据：
Configuring Open vSwitch to be managed by OpenDaylight
On each host (both compute and control nodes) we will clear the pre-existing Open vSwitch
config and set OpenDaylight to manage the switch:
? Stop the Open vSwitch service, and clear existing OVSDB (ODL expects to manage vSwitches completely)
# systemctl stop openvswitch
# rm -rf /var/log/openvswitch/*
# rm -rf /etc/openvswitch/conf.db
# systemctl start openvswitch


At this stage, your Open vSwitch configuration should be empty:
[root@dneary-odl-compute2 ~]# ovs-vsctl show
9f3b38cb-eefc-4bc7-828b-084b1f66fbfd
ovs_version: "2.1.3"




步骤3：配置openvswitch


注意7：此时应该在所有的节点执行下面的命令，否则创建网络后，br-int中的vxlan/gre口没有创建，karaf.log中报
"Tunnel end-point configuration missing. Please configure it in OpenVSwitch Table. Check source null or destination null"
其中ovs_local_ip是用于连接虚拟机之间数据通信的物理接口的ip（也就是原来/etc/neutron/plugin.ini中的local_ip),这个ip用于创建vxlan接口，让不同主机上的同一子网的虚拟机通信；


OVSUUID=$(ovs-vsctl get Open_vSwitch . _uuid);ovs-vsctl set Open_vSwitch $OVSUUID other_config:local_ip="ovs_local_ip"


设置所有节点的openflow交换机（ovs)对应的SDN控制器：
Set OpenDaylight as the manager on all nodes
# ovs-vsctl set-manager tcp:${CONTROL_HOST}:6640


You should now see a new section in your Open vSwitch configuration showing that you
are connected to the OpenDaylight server, and OpenDaylight will automatically create a br-int bridge:
[root@dneary-odl-compute2 ~]# ovs-vsctl show
9f3b38cb-eefc-4bc7-828b-084b1f66fbfd
Manager "tcp:172.16.21.56:6640"
is_connected: true
Bridge br-int
Controller "tcp:172.16.21.56:6633"
fail_mode: secure
Port br-int
Interface br-int
ovs_version: "2.1.3"


(BUG WORKAROUND) If SELinux is enabled, you may not have a security context in place
which allows Open vSwitch remote administration. If you do not see the result above
(specifically, if you do not see "is_connected: true" in the Manager section), set SELinux to
Permissive mode on all nodes and ensure it stays that way after boot:
# setenforce 0
# sed -i -e 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/selinux/config
注意8：如果br-int还是没有，参考注意4；
配置完后，创建虚拟机后应该能看到vxlan接口：
[root@nova1 ~]# ovs-vsctl show
1fe3d90f-380f-4421-abf7-c942b3ec92b7
    Manager "tcp:192.168.88.162:6640"
        is_connected: true
    Bridge br-int
        Controller "tcp:192.168.88.162:6653"
            is_connected: true
        fail_mode: secure
        Port "tap4f18a7c1-d0"
            Interface "tap4f18a7c1-d0"
        Port "vxlan-10.0.1.32"
            Interface "vxlan-10.0.1.32"
                type: vxlan
                options: {key=flow, local_ip="10.0.1.31", remote_ip="10.0.1.32"}
        Port "vxlan-10.0.1.33"
            Interface "vxlan-10.0.1.33"
                type: vxlan
                options: {key=flow, local_ip="10.0.1.31", remote_ip="10.0.1.33"}


[root@nova1 ~]# ovs-ofctl dump-ports-desc br-int -OopenFlow13;ovs-ofctl dump-flows br-int -OopenFlow13
OFPST_PORT_DESC reply (OF1.3) (xid=0x2):
 2(vxlan-10.0.1.33): addr:22:4e:c1:a5:43:2c
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 3(vxlan-10.0.1.32): addr:26:bb:f4:a0:4d:1d
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 4(tap4f18a7c1-d0): addr:fe:16:3e:c6:d7:6a
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:ea:31:e3:31:83:4e
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x0, duration=4091.277s, table=0, n_packets=85, n_bytes=5138, in_port=4,dl_src=fa:16:3e:c6:d7:6a actions=set_field:0x2b69->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
 cookie=0x0, duration=264060.566s, table=0, n_packets=42, n_bytes=2884, priority=0 actions=goto_table:20
 cookie=0x0, duration=4091.275s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=4 actions=drop
 cookie=0x0, duration=4091.258s, table=0, n_packets=47, n_bytes=2030, tun_id=0x2b69,in_port=3 actions=load:0x2->NXM_NX_REG0[],goto_table:20
 cookie=0x0, duration=4091.249s, table=0, n_packets=29, n_bytes=3185, tun_id=0x2b69,in_port=2 actions=load:0x2->NXM_NX_REG0[],goto_table:20
 cookie=0x0, duration=264061.788s, table=0, n_packets=1693, n_bytes=192176, dl_type=0x88cc actions=CONTROLLER:65535
 cookie=0x0, duration=264060.563s, table=20, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:30
 cookie=0x0, duration=264060.555s, table=30, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:40
 cookie=0x0, duration=4091.279s, table=40, n_packets=16, n_bytes=1432, priority=36001,ip,in_port=4,dl_src=fa:16:3e:c6:d7:6a,nw_src=192.168.2.4 actions=goto_table:50
 cookie=0x0, duration=264060.549s, table=40, n_packets=106587, n_bytes=4479637, priority=0 actions=goto_table:50
 cookie=0x0, duration=112926.553s, table=40, n_packets=7, n_bytes=2348, priority=61012,udp,tp_src=68,tp_dst=67 actions=goto_table:50
 cookie=0x0, duration=4091.283s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=4,tp_src=67,tp_dst=68 actions=drop
 cookie=0x0, duration=264060.544s, table=50, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:60
 cookie=0x0, duration=264060.538s, table=60, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:70
 cookie=0x0, duration=264060.533s, table=70, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:80
 cookie=0x0, duration=264060.528s, table=80, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:90
 cookie=0x0, duration=264060.523s, table=90, n_packets=106608, n_bytes=4482691, priority=0 actions=goto_table:100
 cookie=0x0, duration=4091.287s, table=90, n_packets=2, n_bytes=726, priority=61006,udp,dl_src=fa:16:3e:e0:d0:54,tp_src=67,tp_dst=68 actions=goto_table:100
 cookie=0x0, duration=264060.518s, table=100, n_packets=106610, n_bytes=4483417, priority=0 actions=goto_table:110
 cookie=0x0, duration=4091.261s, table=110, n_packets=0, n_bytes=0, priority=8192,tun_id=0x2b69 actions=drop
 cookie=0x0, duration=264060.499s, table=110, n_packets=42, n_bytes=2884, priority=0 actions=drop
 cookie=0x0, duration=4091.268s, table=110, n_packets=57, n_bytes=2838, priority=16384,reg0=0x2,tun_id=0x2b69,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:4
 cookie=0x0, duration=4091.265s, table=110, n_packets=65, n_bytes=3538, priority=16383,reg0=0x1,tun_id=0x2b69,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:4,output:3,output:2
 cookie=0x0, duration=4334.772s, table=110, n_packets=7, n_bytes=552, tun_id=0x2b69,dl_dst=fa:16:3e:e0:d0:54 actions=output:2
 cookie=0x0, duration=4016.152s, table=110, n_packets=10, n_bytes=866, tun_id=0x2b69,dl_dst=fa:16:3e:a9:91:6e actions=output:2
 cookie=0x0, duration=4091.272s, table=110, n_packets=19, n_bytes=2377, tun_id=0x2b69,dl_dst=fa:16:3e:c6:d7:6a actions=output:4
 cookie=0x0, duration=4126.913s, table=110, n_packets=3, n_bytes=182, tun_id=0x2b69,dl_dst=fa:16:3e:e2:aa:65 actions=output:3


注意9：有个odl_tools可以辅助查看ODL的信息：
git checkout https://github.com/shague/odl_tools


[root@controller odl_tools]# /opt/ODL/odl_tools/showOvsdbMdsal.py --port 8080 -c --ip=sdncontrollerip


aliasMap:
  alpha      -> openflow:257499985969998
  bravo      -> openflow:174756858943808
  charlie    -> openflow:227309101726016
  delta      -> ovsdb://uuid/da64fd4b-d077-4d4a-920b-cc43410381bd/bridge/br-int
  echo       -> ovsdb://uuid/6a7bea6d-71c9-42a6-83fa-372b79676cc8/bridge/br-int


Bridges in ovsdb://uuid/1fe3d90f-380f-4421-abf7-c942b3ec92b7
  alpha:br-int controller:tcp:192.168.88.162:6653 connected:None
    of:None vxlan-10.0.1.33
Bridges in ovsdb://uuid/2573ce18-b1c8-4844-93e6-79694d17256d
  bravo:br-int controller:tcp:192.168.88.162:6653 connected:None
    of:None vxlan-10.0.1.33
Bridges in ovsdb://uuid/58cdd569-3eac-4014-90cf-c68839d1a557
  charlie:br-int controller:tcp:192.168.88.162:6653 connected:None
    of:None vxlan-10.0.1.31
Bridges in ovsdb://uuid/6a7bea6d-71c9-42a6-83fa-372b79676cc8
  echo:br-int controller:tcp:192.168.88.162:6653 connected:None
Bridges in ovsdb://uuid/da64fd4b-d077-4d4a-920b-cc43410381bd
  delta:br-int controller:tcp:192.168.88.162:6653 connected:None


config tree flows at alpha
  table 0: DEFAULT_PIPELINE_FLOW_0
  table 0: DropFilter_4
  table 0: LLDP
  table 0: LocalMac_11113_4_fa:16:3e:c6:d7:6a




Make sure all nodes, including the control node, are connected to OpenDaylight
If you reload DLUX, you should now see that all of your Open vSwitch nodes are now connected to OpenDaylight


If something has gone wrong, check <code>data/log/karaf.log</code> under the
OpenDaylight distribution directory. If you do not see any interesting log entries, set
logging for OVSDB to TRACE level inside Karaf and try again:
log:set TRACE ovsdb




步骤4：配置neutron
Configuring Neutron to use OpenDaylight
Once you have configured the vSwitches to connect to OpenDaylight, you can now ensure
that OpenStack Neutron is using OpenDaylight.
First, ensure that port 8080 (which will be used by OpenDaylight to listen for REST calls)
is available. By default, swift-proxy-service listens on the same port, and you may need
to move it (to another port or another host), or disable that service. I moved it to port
8081 by editing <code>/etc/swift/proxy-server.conf</code> and <code>/etc/cinder/
cinder.conf</code>, modifying iptables appropriately, and restarting swift-proxy-service
and OpenDaylight.


Configure Neutron to use OpenDaylight’s ML2 driver:
crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers opendaylight
crudini --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan
cat <<EOT>> /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2_odl]
password = admin
username = admin
url = http://${CONTROL_HOST}:8080/controller/nb/v2/neutron
EOT


Reset Neutron’s ML2 database
mysql -e "drop database if exists neutron_ml2;"
mysql -e "create database neutron_ml2 character set utf8;"
mysql -e "grant all on neutron_ml2.* to 'neutron'@'%';"
neutron-db-manage --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head


Restart neutron-server:
systemctl start neutron-server


Verifying it works
Verify that OpenDaylight’s ML2 interface is working:
curl -u admin:admin http://${CONTROL_HOST}:8080/controller/nb/v2/neutron/networks
{
"networks" : [ ]
}




If this does not work or gives an error, check Neutron’s log file in <code>/var/log/
neutron/server.log</code>. Error messages here should give some clue as to what the
problem is in the connection with OpenDaylight




下面这些命令也可以通过在页面操作来实现，创建网络/子网/虚拟机/路由器，不同的虚拟机保证在不同的主机上，看虚拟机之间/虚拟机-router之间是否可以通信；
Create a net, subnet, router, connect ports, and start an instance using the Neutron CLI:
neutron router-create router1
neutron net-create private
neutron subnet-create private --name=private_subnet 10.10.5.0/24
neutron router-interface-add router1 private_subnet
nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test1
nova boot --flavor <flavor> --image <image id> --nic net-id=<network id> test2
At this point, you have confirmed that OpenDaylight is creating network end-points for instances on your network and managing traffic to them.




Congratulations! You’re done!