[OpenStack] Linuxbridge 的Permission Denied故障调查

同事发现compute节点无法创建虚拟机：调查发现是因为compute网络断开。

从ipmi进入compute节点查看情况，发现是物理网卡对应的linuxbridge的mac地址不对，全部为0，如下图所示

名为brqd8***的linuxbridge mac地址全为00

使用brctl show 可以确认，brqd8***这个网桥添加物理网卡（eno1）失败，所以导致mac地址为00

继续查看/var/log/neutron/linuxbridge-agent.log，发现报错

2020-12-15 15:06:47.559 2194 INFO neutron.plugins.ml2.drivers.agent._common_agent [-] Stopping Linux bridge agent agent.
2020-12-15 15:06:47.566 2194 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=2798
2020-12-15 15:06:49.441 3108 INFO neutron.common.config [-] Logging enabled!
2020-12-15 15:06:49.441 3108 INFO neutron.common.config [-] /usr/bin/neutron-linuxbridge-agent version 15.3.0
2020-12-15 15:06:49.442 3108 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface mappings: {'provider': 'eno1'}
2020-12-15 15:06:49.442 3108 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Bridge mappings: {}
2020-12-15 15:06:49.443 3108 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/neutron/neutron-dist.conf', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/linuxbridge_agent.ini', '--config-dir', '/etc/neutron/conf.d/neutron-linuxbridge-agent', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpJwmauH/privsep.sock']
2020-12-15 15:06:50.343 3108 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
2020-12-15 15:06:50.273 3138 INFO oslo.privsep.daemon [-] privsep daemon starting
2020-12-15 15:06:50.280 3138 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2020-12-15 15:06:50.284 3138 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none
2020-12-15 15:06:50.285 3138 INFO oslo.privsep.daemon [-] privsep daemon running as pid 3138
2020-12-15 15:06:51.114 3108 INFO oslo_rootwrap.client [-] Spawned new rootwrap daemon process with pid=3154
2020-12-15 15:06:51.175 3108 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Agent initialized successfully, now running... 
2020-12-15 15:06:51.252 3108 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] RPC agent_id: lb3cecef3fb068
2020-12-15 15:06:51.257 3108 INFO neutron.agent.agent_extensions_manager [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Loaded agent extensions: []
2020-12-15 15:06:51.556 3108 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Linux bridge agent Agent RPC Daemon Started!
2020-12-15 15:06:51.557 3108 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Linux bridge agent Agent out of sync with plugin!
2020-12-15 15:06:51.567 3108 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Clearing orphaned ARP spoofing entries for devices []
2020-12-15 15:06:51.576 3108 INFO neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Clearing orphaned ARP spoofing entries for devices []
2020-12-15 15:17:54.031 3108 INFO neutron.agent.securitygroups_rpc [req-12652cec-309c-4878-bba5-0d970fbc702c 5e950505d8db4d259185b38ff3e00036 cb737568eee34b8fab31a4538da54004 - - -] Security group member updated [u'daf22ee8-9a32-4d96-8eb5-73d328111c91']
2020-12-15 15:17:57.572 3108 INFO neutron.agent.securitygroups_rpc [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Preparing filters for devices set(['tap6f4fd167-58'])
2020-12-15 15:17:57.635 3108 WARNING oslo_rootwrap.client [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Leaving behind already spawned process with pid 3154, root should kill it if it's still there (I can't): error: [Errno 32] Broken pipe
2020-12-15 15:17:57.940 3108 INFO oslo_rootwrap.client [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Spawned new rootwrap daemon process with pid=3433
2020-12-15 15:17:58.356 3108 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Port tap6f4fd167-58 updated. Details: {u'profile': {}, u'network_qos_policy_id': None, u'propagate_uplink_status': False, u'qos_policy_id': None, u'allowed_address_pairs': [], u'admin_state_up': True, u'network_id': u'd8eab5c8-9f09-4248-bf69-9aba07ec3ef8', u'segmentation_id': None, u'mtu': 1500, u'device_owner': u'compute:nova', u'physical_network': u'provider', u'mac_address': u'fa:16:3e:a9:55:00', u'device': u'tap6f4fd167-58', u'port_security_enabled': True, u'port_id': u'6f4fd167-583e-481e-937a-484f60a7814b', u'fixed_ips': [{u'subnet_id': u'f0b09cd8-20e3-48ff-81f7-2e7d15cac445', u'ip_address': u'192.168.100.165'}], u'network_type': u'flat'}
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Error in agent loop. Devices info: {'current': set(['tap6f4fd167-58']), 'timestamps': {'tap6f4fd167-58': 7}, 'removed': set([]), 'added': set(['tap6f4fd167-58']), 'updated': set([])}: NetlinkError: (13, 'Permission denied')
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     resync_a = self.treat_devices_added_updated(devices_added_updated)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._process_device_if_exists(device_details)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, device_details['device_owner'])
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 587, in plug_interface
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     network_segment.mtu)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 523, in add_tap_interface
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return False
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.force_reraise()
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     six.reraise(self.type_, self.value, self.tb)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 515, in add_tap_interface
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     tap_device_name, device_owner, mtu)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 548, in _add_tap_interface
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     mtu):
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 499, in ensure_physical_in_bridge
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     physical_interface)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 287, in ensure_flat_bridge
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     if self.ensure_bridge(bridge_name, physical_interface):
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 457, in ensure_bridge
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.update_interface_ip_details(bridge_name, interface)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 419, in update_interface_ip_details
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     gateway)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 395, in _update_interface_ip_details
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     dst_device.addr.add(cidr=ip['cidr'])
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 528, in add
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     add_broadcast)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 802, in add_ip_address
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, namespace, scope, broadcast)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 73, in sync_inner
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return input_func(*args, **kwargs)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return self.channel.remote_call(name, args, kwargs)
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise exc_type(*result[2])
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent NetlinkError: (13, 'Permission denied')
2020-12-15 15:17:58.625 3108 ERROR neutron.plugins.ml2.drivers.agent._common_agent 
2020-12-15 15:17:59.571 3108 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-acbe8bbd-551e-45c1-b156-74876f2c9d6a - - - - -] Linux bridge agent Agent out of sync with plugin!

接着g资料，翻了n多网页后，发现一篇文章说可能是ipv6导致（原文地址：https://www.mail-archive.com/yahoo-eng-team@lists.launchpad.net/msg84024.html）

抱着试试看的心态禁用ipv6

编辑/etc/sysctl.conf添加下列两行

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

然后执行sysctl -p

重启后再次创建instance，成功！