[Backport][bug/1352826] applying iptables rules takes too long when large scale deployment
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Mirantis OpenStack |
Critical
| Alexander Ignatov | ||
| 4.1.x |
Critical
| MOS Neutron | ||
| 5.0.x |
Critical
| MOS Neutron | ||
| 5.1.x |
Critical
| Alexander Ignatov | ||
| 6.0.x |
Critical
| Alexander Ignatov |
Bug Description
This is a backport of https:/
Original description
================
I found the time to finishing the applying iptables rules( in neutron/
This will lead that the time of bringing new created port up when booting an instance will take very long, and if the vif_plugging_
Although optimization on _modify_rules in patch https:/
Further optimazation on _modify_rules need be done to fit the situation of Large-scale deployment.
Changed in mos: | |
status: | Triaged → In Progress |
OSCI Robot (oscirobot) wrote on 2014-12-04: | #1 |
OSCI Robot (oscirobot) wrote on 2014-12-04: | #2 |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created
Files placed on repository:
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
python-
NOTE: Changeset is not merg...
tags: | added: scale |
Changed in mos: | |
importance: | High → Critical |
Nastya Urlapova (aurlapova) wrote on 2014-12-05: | #3 |
Hm, did you make the same loading as for 5.1.1, so 6.0 - more than 980 vms? Let me remind you that we certified 5.1.1 only on 20 nodes. Aleksander, please clarify this moment for 5.1.1 version.
Alexander Ignatov (aignatov) wrote on 2014-12-06: | #4 |
Nastya, this is original description from the bug filed in upstream.
The RCA of this bug is algorithm for modifying iptables rules on each compute. So the main factor of this issue is a number of VMs per compute and number of security groups rules.
If we read comments from upstream issue we can find that bug submitter talks about 100 VMs per compute approximately:
"We have got 11 hosts, one is the controller, other 10 are compute nodes." (c)
So I did several checks to verify the patch:
TestCase1 (measuring a time of updating iptables rules):
Prerequisites:
- ~500 rules in security group "test-sg" to attach for VMs
- already deployed 96 VMs on the same compute using an availability-zone which contains the same compute for each VM
- flavor is not important here, cirros image
1. When all pre-deployed VMs are Active check that openvswich-agent doesn't consume 100% CPU
2. Start booting 1 new VM with "test-sg" security-group and start measuring time when ovs-agent consumes 100% CPU
Expected result: 100% CPU usage for ~10 sec, VM is booted success
Note: before this fix you could check that VM is being booted too long (8-10 mins and could run into Error state) and CPU about 100% for the whole period of booting
Second check is simple:
1. Deploy 10 VMs on dedicated compute, attach "test-sg" for each instance.
2. Deploy 20 VMs on dedicated compute, attach "test-sg" for each instance.
3 Deploy 50 VMs on dedicated compute, attach "test-sg" for each instance.
Expected result: all 80 VMs are up and runnig
Note: before fix number of failed VMs was about 90%-30%
All my tests are passed I assume that this bug is fixed correctly and proposing to merge it.
Alexander Ignatov (aignatov) wrote on 2014-12-06: | #5 |
Also I've performed the following check using WARNING logs in python at the beginning and ending of function which modifies security group rules using "START MODIFY RULES" and "END MODIFY RULES" labels:
After patch:
There was booted 96 VMs and 460 rules per VM
Adding +1 VM
2014-12-05 22:51:46.616 22962 WARNING neutron.
2014-12-05 22:51:57.647 22962 WARNING neutron.
Result: 11 sec
Adding +1 VM
2014-12-05 22:55:35.525 22962 WARNING neutron.
2014-12-05 22:55:46.934 22962 WARNING neutron.
Result: 11 sec
After deleting 4VMs
2014-12-05 23:01:21.946 22962 WARNING neutron.
2014-12-05 23:01:33.097 22962 WARNING neutron.
Result: 11 sec
Before patch
There was booted 96 VMs and 460 rules per VM
2014-12-05 23:28:30.195 30024 WARNING neutron.
2014-12-05 23:38:38.574 30024 WARNING neutron.
Result: 10 mins
Booted 1 VM and
2014-12-05 23:47:12.682 30024 WARNING neutron.
2014-12-05 23:57:13.310 30024 WARNING neutron.
Result: operation of modifying iptables took 10 min and VM moved into Error state
Deleting of 20 VMs
2014-12-06 00:00:18.216 30024 WARNING neutron.
2014-12-06 00:10:04.423 30024 WARNING neutron.
Result: operation of modifying iptables took 10 min
Alexander Ignatov (aignatov) wrote on 2014-12-06: | #6 |
I think patch is ready to be merged
OSCI Robot (oscirobot) wrote on 2014-12-06: | #7 |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0.mira13
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged
Files placed on repository:
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
Changeset merged. Package placed on primary repository
RPM repository URL: http://
OSCI Robot (oscirobot) wrote on 2014-12-06: | #8 |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.2, package release == fuel6.0~mira12
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged
Files placed on repository:
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
python-
Changeset merged. Package placed on primary repository
DEB repository URL: http://
Mike Scherbakov (mihgen) wrote on 2014-12-07: | #9 |
Colleagues, why is it Incomplete for 5.1.2? I thought it affects 5.1.X. Let's figure out: if it does not, then the status of this bug should be Invalid for 5.1.1 / 5.1.2.
OSCI Robot (oscirobot) wrote on 2014-12-11: | #10 |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created
Files placed on repository:
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://
OSCI Robot (oscirobot) wrote on 2014-12-11: | #11 |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created
Files placed on repository:
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
OSCI Robot (oscirobot) wrote on 2014-12-15: | #12 |
RPM package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2.mira2
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged
Files placed on repository:
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
Changeset merged. Package placed on primary repository
RPM repository URL: http://
OSCI Robot (oscirobot) wrote on 2014-12-15: | #13 |
DEB package neutron has been built for project openstack/neutron
Package version == 2014.1.3, package release == fuel5.1.2~mira2
Changeset: https:/
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: change-merged
Files placed on repository:
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
neutron-
python-
Changeset merged. Package placed on primary repository
DEB repository URL: http://
Alexander Ignatov (aignatov) wrote on 2014-12-22: | #16 |
Release note for 5.1.1:
ISSUE: Virtual Machines fail to start due to slow processing of IP tables rules
SUMMARY: In some larger scale environments, virtual machines fail to launch due to slow processing of IP tables rules. Environments with large numbers of compute nodes and VMs can lead to large IP tables in total if they apllied. Slow processing of large IP tables can lead to service timeouts that may prevent virtual machines from launching.
The problem is fixed in python-neutron package from MOS 5.1.2 (attached to the LP ticket).
Steps to apply patches on CentOS environments:
(should be done one by one on all compute nodes, then on all controllers, also one by one)
Upload package python-
Execute the following on the node:
Run the following commands:
[root@node-2 ~]# rpm -Uvh --nodeps python-
warning: python-
Preparing... #######
1:python-neutron #######
Make sure step 2 doesn’t produce any errors
On computes nodes do: service neutron-
On one of controller nodes: crm resource restart clone_p_
Dmitry Borodaenko (dborodaenko) wrote on 2014-12-29: | #18 |
If it was a Neutron bug, how is it possible that this had to be fixed in 5.1.x but doesn't reproduce in 5.0.x?
Dmitry Borodaenko (dborodaenko) wrote on 2014-12-29: | #19 |
We also still need a backport of this for 4.1.x, status changed from Won't Fix to Confirmed.
Dmitry Borodaenko (dborodaenko) wrote on 2015-01-26: | #20 |
No updates since December, what's the status of the backports?
Alexander Ignatov (aignatov) wrote on 2015-01-29: | #21 |
For this patch which affects 4.x and 5.x versions Mirantis published official Tech Bulletin:
Consider to move this bug to Won't fix.
RPM package neutron has been built for project openstack/neutron mira13. git.f90d4c0. a299407
Package version == 2014.2, package release == fuel6.0.
Changeset: https:/ /review. fuel-infra. org/1237 ci/fuel- 6.0/2014. 2
project: openstack/neutron
branch: openstack-
author: Alexander Ignatov
committer: Alexander Ignatov
subject: Improve the performance of _modify_rules() in IptablesManager
status: patchset-created
Files placed on repository: neutron- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- bigswitch- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- brocade- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- cisco-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- hyperv- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- ibm-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- linuxbridge- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- mellanox- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- metaplugin- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- metering- agent-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- midonet- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- ml2-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- nec-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- nuage-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- ofagent- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- oneconvergence- nvsd-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- opencontrail- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- openvswitch- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- plumgrid- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- ryu-2014. 2-fuel6. 0.mira13. git.f90d4c0. a299407. noarch. rpm neutron- vmware- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- vpn-agent- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm neutron- 2014.2- fuel6.0. mira13. git.f90d4c0. a299407. noarch. rpm
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
NOTE: Changeset is not merged, created temporary package repository. osci-obs. vm.mirantis. net:82/ centos- fuel-6. 0-stable- 1237/centos
RPM repository URL: http://