OpenStack Folsom实例重启之后cinder-volume服务无法启动的解决办法

Cinder组件装在Controller节点，节点发生故障重启后，发现挂载了volumn的实例无法启动，错误日志如下：

2013-06-28 09:48:46 14338 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kw)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     pass
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     kwargs['instance']['uuid'], e, sys.exc_info())
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 191, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 970, in start_instance
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.power_on_instance(context, instance)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kw)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 176, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     pass
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 162, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 197, in decorated_function
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp     kwargs['instance']['uuid'], e, sys.exc_info())
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp KeyError: 'instance'
2013-06-28 09:48:46 14338 TRACE nova.openstack.common.rpc.amqp

造成的原因分析：应该是实例VM在启动时，无法挂载虚拟磁盘volumn，查看cinder状态，发现果然不能创建及挂载volumn，错误日志如下：

2013-05-10 14:12:22 2235 CRITICAL cinder [-] Got unknown keyword args to utils.execute: {'old_name': None}
2013-05-10 14:12:22 2235 TRACE cinder Traceback (most recent call last):
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/bin/cinder-volume", line 48, in 
2013-05-10 14:12:22 2235 TRACE cinder service.wait()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 422, in wait
2013-05-10 14:12:22 2235 TRACE cinder _launcher.wait()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 127, in wait
2013-05-10 14:12:22 2235 TRACE cinder service.wait()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2013-05-10 14:12:22 2235 TRACE cinder return self._exit_event.wait()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-05-10 14:12:22 2235 TRACE cinder return hubs.get_hub().switch()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2013-05-10 14:12:22 2235 TRACE cinder return self.greenlet.switch()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2013-05-10 14:12:22 2235 TRACE cinder result = function(*args, **kwargs)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 88, in run_server
2013-05-10 14:12:22 2235 TRACE cinder server.start()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 159, in start
2013-05-10 14:12:22 2235 TRACE cinder self.manager.init_host()
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 101, in init_host
2013-05-10 14:12:22 2235 TRACE cinder self.driver.ensure_export(ctxt, volume)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 352, in ensure_export
2013-05-10 14:12:22 2235 TRACE cinder old_name=old_name)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 198, in create_iscsi_target
2013-05-10 14:12:22 2235 TRACE cinder self._new_target(name, tid, **kwargs)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 211, in _new_target
2013-05-10 14:12:22 2235 TRACE cinder **kwargs)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/iscsi.py", line 60, in _run
2013-05-10 14:12:22 2235 TRACE cinder self._execute(self._cmd, *args, run_as_root=True, **kwargs)
2013-05-10 14:12:22 2235 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 146, in execute
2013-05-10 14:12:22 2235 TRACE cinder 'to utils.execute: %r') % kwargs)
2013-05-10 14:12:22 2235 TRACE cinder Error: Got unknown keyword args to utils.execute: {'old_name': None}
2013-05-10 14:12:22 2235 TRACE cinder

错误分析：这是OpenStack Folsom版本的一个BUG，造成cinder-volumn服务在节点重启后无法正常启动，最新版本已经修订 https://bugs.launchpad.net/cinder/+bug/1175207，可以自己通过修改源代码进行修改，修改如下：

1、打开文件/usr/lib/python2.7/dist-packages/cinder/volume/driver.py

2、找到352行，注释掉old_name=old_name这行代码，修改如下：

        self.tgtadm.create_iscsi_target(iscsi_name, iscsi_target,
                                        0, volume_path,
                                        check_exit_code=False)
                                        #old_name=old_name)

3、重启cinder-volumn服务

service cinder-volume restart

服务正常启动，问题解决。