nova-cinder交互流程分析

http://aspirer2004.blog.163.com/blog/static/106764720134755131463/

本文主要调研cinder与nova的交互流程，分析了自有块存储系统与nova的整合问题。

1.   Nova现有API统计

nova已经支持的块设备API可以参考http://api.openstack.org/api-ref.html中VolumeAttachments，Volume Extension to Compute两个部分的说明。

操作类（所有删除操作都是异步的，需要用户自行调用查询API进行确认）：

1)        创建块设备（包括从快照恢复出块设备）（可以指定块设备AZ）（需要提供用户ID）

2)        删除块设备（需要提供用户ID和块设备ID）

3)        挂载块设备（需要指定用户ID，云主机ID，块设备ID）

4)        卸载块设备（需要指定用户ID，云主机ID，块设备ID）

5)        给块设备建快照（需要提供用户ID和块设备ID）

6)        删除快照（需要提供用户ID和快照ID）

查询类：

1)        列出云主机上挂载的块设备（需要指定用户ID和云主机ID）

2)        根据云主机ID及挂载在其上的块设备ID查询挂载详细信息（需要指定用户ID，云主机ID，块设备ID）

3)        查询用户所有的块设备（需要提供用户ID）

4)        根据块设备ID查询用户某个块设备的详细信息（需要提供用户ID和块设备ID）

5)        查询用户所有的块设备快照（需要提供用户ID）

6)        查询用户所有的块设备快照详细信息（需要提供用户ID和快照ID）

 

需要新增API：

1)        扩容API（我们这边有新增API的经验，比较容易实现）

2.   Nova-Cinder交互流程分析

这里只选择两个比较典型的交互过程进行分析。

2.1    创建块设备cinder流程

创建块设备支持从快照恢复出块设备。

API URL：POSThttp://localhost:8774/v1.1/{tenant_id}/os-volumes

    Request parameters

    Parameter        Description

    tenant_id                   Theunique identifier of the tenant or account.

    volume_id         The unique identifier for a volume.

    Volume              A partial representation of avolume that is used to create a volume.

         Create Volume Request: JSON

    {

        "volume":{

            "display_name":"vol-001",

            "display_description":"Another volume.",

            "size":30,

            "volume_type":"289da7f8-6440-407c-9fb4-7db01ec49164",

            "metadata":{"contents":"junk"},

            "availability_zone":"us-east1"

         }

    }

 

    Create Volume Response: JSON

    {

        "volume":{

            "id":"521752a6-acf6-4b2d-bc7a-119f9148cd8c",

            "display_name":"vol-001",

            "display_description":"Another volume.",

            "size":30,

            "volume_type":"289da7f8-6440-407c-9fb4-7db01ec49164",

            "metadata":{"contents":"junk"},

            "availability_zone":"us-east1",

            "snapshot_id": null,

            "attachments":[],

            "created_at":"2012-02-14T20:53:07Z"

         }

    }

 

 

# nova\api\openstack\compute\contrib\volumes.py:

VolumeController.create()

    @wsgi.serializers(xml=VolumeTemplate)

    @wsgi.deserializers(xml=CreateDeserializer)

    def create(self, req, body):

        """Creates a newvolume."""

        context = req.environ['nova.context']

        authorize(context)

 

        if not self.is_valid_body(body,'volume'):

            raise exc.HTTPUnprocessableEntity()

 

        vol = body['volume']

        # 卷类型，暂时不支持，参数不传入即可

        vol_type = vol.get('volume_type',None)

        if vol_type:

            try:

                vol_type = volume_types.get_volume_type_by_name(context,

                                                               vol_type)

            except exception.NotFound:

                raise exc.HTTPNotFound()

 

        metadata = vol.get('metadata',None)

        # 如果要从快照恢复卷，传入要被恢复的快照ID即可

        snapshot_id = vol.get('snapshot_id')

 

        if snapshot_idis not None:

            # 从快照恢复云硬盘需要实现如下方法，self.volume_api下面会有说明

            snapshot = self.volume_api.get_snapshot(context, snapshot_id)

        else:

            snapshot = None

 

        size = vol.get('size',None)

        if size is None and snapshot is not None:

            size = snapshot['volume_size']

 

        LOG.audit(_("Create volume of %s GB"), size, context=context)

        # 卷AZ信息

        availability_zone = vol.get('availability_zone',None)

        # 云硬盘需要实现如下方法，self.volume_api下面会有说明

        new_volume = self.volume_api.create(context,

                                           size,

                                            vol.get('display_name'),

                                            vol.get('display_description'),

                                           snapshot=snapshot,

                                           volume_type=vol_type,

                                           metadata=metadata,

                                           availability_zone=availability_zone

                                           )

 

        # TODO(vish): Instance should be None at dblayer instead of

        #            trying to lazy load, but for now we turn it into

        #            a dict to avoid an error.

        retval = _translate_volume_detail_view(context, dict(new_volume))

        result = {'volume': retval}

 

        location ='%s/%s' %(req.url, new_volume['id'])

 

        return wsgi.ResponseObject(result, headers=dict(location=location))

 

    # self.volume_api说明

    self.volume_api= volume.API()

    volume是from novaimport volume导入的

    # nova\volume\__init__.py:

    def API():

        importutils = nova.openstack.common.importutils

        cls = importutils.import_class(nova.flags.FLAGS.volume_api_class)

        return cls()

可见self.volume_api调用的所有方法都是由配置项volume_api_class决定的，默认配置是使用nova-volume的API封装类，

   cfg.StrOpt('volume_api_class',

             default='nova.volume.api.API',

             help='The fullclass name of the volume API class to use'),

    也可以改用cinder的API封装类，只要把配置改为volume_api_class=nova.volume.cinder.API即可，cinder API封装类通过调用封装了创建卷方法的cinder_client库来调用到cinder的API，云硬盘可以实现一个类似的client库，也可以直接调用已有的API来实现相同的动作（cinder_client库也是对cinder API调用的封装），云硬盘可以参考nova\volume\cinder.py开发自己的API封装类，供NVS使用，由于API已经开发完成，所以只是封装API，工作量应该不是很大，需要注意的应该是认证问题。

    快照相关操作及查询与上述流程没有区别，只要模仿nova\volume\cinder.py即可实现。

 

 

2.2    挂载块设备cinder流程

    API URL：POSThttp://localhost:8774/v2/{tenant_id}/servers/{server_id}/os-volume_attachments

   

    Request parameters

    Parameter        Description

    tenant_id                   TheID for the tenant or account in a multi-tenancy cloud.

    server_id          The UUID for the server of interest toyou.

    volumeId          IDof the volume to attach.

    device                Nameof the device e.g. /dev/vdb. Use "auto" for autoassign (ifsupported).

    volumeAttachment          Adictionary representation of a volume attachment.

Attach Volume to Server Request: JSON

    {

        'volumeAttachment':{

          'volumeId': volume_id,

          'device': device

        }

    }

 

    Attach Volume to Server Response: JSON

    {

        "volumeAttachment":{

            "device":"/dev/vdd",

            "serverId":"fd783058-0e27-48b0-b102-a6b4d4057cac",

            "id":"5f800cf0-324f-4234-bc6b-e12d5816e962",

            "volumeId":"5f800cf0-324f-4234-bc6b-e12d5816e962"

        }

    }

需要注意的是这个API返回是同步的，但挂载卷到虚拟机是异步的。

# nova\api\openstack\compute\contrib\volumes.py:

VolumeAttachmentController.create()

    @wsgi.serializers(xml=VolumeAttachmentTemplate)

    def create(self, req, server_id, body):

        """Attach a volume to aninstance."""

        context = req.environ['nova.context']

        authorize(context)

 

        if not self.is_valid_body(body,'volumeAttachment'):

            raise exc.HTTPUnprocessableEntity()

 

        volume_id = body['volumeAttachment']['volumeId']

        device = body['volumeAttachment'].get('device')

 

        msg = _("Attach volume%(volume_id)s to instance %(server_id)s"

                " at %(device)s")% locals()

        LOG.audit(msg, context=context)

 

        try:

            instance = self.compute_api.get(context, server_id)

            # nova-compute负责挂载卷到虚拟机

            device = self.compute_api.attach_volume(context, instance,

                                                   volume_id, device)

        except exception.NotFound:

            raise exc.HTTPNotFound()

 

        # The attach is async

        attachment = {}

        attachment['id']= volume_id

        attachment['serverId']= server_id

        attachment['volumeId']= volume_id

        attachment['device']= device

 

        # NOTE(justinsb): And now, we have aproblem...

        # The attach is async, so there's a window inwhich we don't see

        # the attachment (until the attachmentcompletes).  We could also

        # get problems with concurrent requests.  I think we need an

        # attachment state, and to write to the DBhere, but that's a bigger

        # change.

        # For now, we'll probably have to rely onlibraries being smart

 

        # TODO(justinsb): How do I return"accepted" here?

        return {'volumeAttachment': attachment}

   

    # nova\compute\api.py:API.attach_volume()

    @wrap_check_policy

    @check_instance_lock

    def attach_volume(self, context, instance, volume_id, device=None):

        """Attach an existingvolume to an existing instance."""

        # NOTE(vish): Fail fast if the device is notgoing to pass. This

        #            will need to be removed along with the test if we

        #            change the logic in the manager for what constitutes

        #            a valid device.

        if deviceand not block_device.match_device(device):

            raise exception.InvalidDevicePath(path=device)

        # NOTE(vish): This is done on the computehost because we want

        #            to avoid a race where two devices are requested at

        #            the same time. When db access is removed from

        #            compute, the bdm will be created here and we will

        #            have to make sure that they are assigned atomically.

        device = self.compute_rpcapi.reserve_block_device_name(

            context, device=device, instance=instance)

        try:

            # 云硬盘需要实现的方法，也可以参考nova\volume\cinder.py

            volume = self.volume_api.get(context, volume_id)

            # 检测卷是否可以挂载

            self.volume_api.check_attach(context, volume)

            # 预留要挂载的卷，防止并发挂载问题

            self.volume_api.reserve_volume(context, volume)

            # RPC Cast异步调用到虚拟机所在的宿主机的nova-compute服务进行挂载

            self.compute_rpcapi.attach_volume(context, instance=instance,

                    volume_id=volume_id, mountpoint=device)

        except Exception:

            with excutils.save_and_reraise_exception():

                self.db.block_device_mapping_destroy_by_instance_and_device(

                        context, instance['uuid'], device)

        # API在这里返回

        return device

 

    # nova\compute\manager.py:ComputeManager.attach_volume()

    @exception.wrap_exception(notifier=notifier, publisher_id=publisher_id())

    @reverts_task_state

    @wrap_instance_fault

    def attach_volume(self, context, volume_id, mountpoint, instance):

        """Attach a volume to aninstance."""

        try:

            return self._attach_volume(context, volume_id,

                                      mountpoint, instance)

        except Exception:

            with excutils.save_and_reraise_exception():

           self.db.block_device_mapping_destroy_by_instance_and_device(

                        context, instance.get('uuid'), mountpoint)

 

    def _attach_volume(self, context, volume_id, mountpoint, instance):

        # 同上面的volume_api.get方法

        volume = self.volume_api.get(context, volume_id)

        context = context.elevated()

        LOG.audit(_('Attaching volume %(volume_id)s to%(mountpoint)s'),

                  locals(), context=context, instance=instance)

        try:

            # 这里返回的是initiator信息，下面有分析

            connector = self.driver.get_volume_connector(instance)

            # 云硬盘需要实现的方法，下面有cinder的具体实现

            connection_info = self.volume_api.initialize_connection(context,

                                                                   volume,

                                                                   connector)

        except Exception: # pylint: disable=W0702

            with excutils.save_and_reraise_exception():

                msg = _("Failed toconnect to volume %(volume_id)s "

                        "while attaching at%(mountpoint)s")

                LOG.exception(msg% locals(), context=context,

                              instance=instance)

                # 这个方法也要实现

                self.volume_api.unreserve_volume(context, volume)

 

        if 'serial' not in connection_info:

            connection_info['serial']= volume_id

 

        try:

            self.driver.attach_volume(connection_info,

                                      instance['name'],

                                     mountpoint)

        except Exception: # pylint: disable=W0702

            with excutils.save_and_reraise_exception():

                msg = _("Failed toattach volume %(volume_id)s "

                        "at %(mountpoint)s")

                LOG.exception(msg% locals(), context=context,

                              instance=instance)

                self.volume_api.terminate_connection(context,

                                                    volume,

                                                    connector)

        # 这个方法也要实现，作用是更新cinder数据库中的卷的状态

        self.volume_api.attach(context,

                               volume,

                               instance['uuid'],

                               mountpoint)

        values = {

            'instance_uuid': instance['uuid'],

            'connection_info': jsonutils.dumps(connection_info),

            'device_name': mountpoint,

            'delete_on_termination':False,

            'virtual_name':None,

            'snapshot_id':None,

            'volume_id': volume_id,

            'volume_size':None,

            'no_device':None}

        self.db.block_device_mapping_update_or_create(context, values)

 

    #nova\virt\libvirt\driver.py:LibvirtDriver.get_volume_connector()

    def get_volume_connector(self, instance):

        if not self._initiator:

            self._initiator= libvirt_utils.get_iscsi_initiator()

            if not self._initiator:

                LOG.warn(_('Could not determine iscsi initiator name'),

                         instance=instance)

        return {

            'ip': FLAGS.my_ip,#宿主机IP地址

            'initiator': self._initiator,

            'host': FLAGS.host#宿主机名

        }

    # nova\virt\libvirt\utils.py:get_iscsi_initiator()

    def get_iscsi_initiator():

        """Get iscsi initiatorname for this machine"""

        # NOTE(vish) openiscsi stores initiator namein a file that

        #           needs root permission to read.

        contents = utils.read_file_as_root('/etc/iscsi/initiatorname.iscsi')

        for l in contents.split('\n'):

            if l.startswith('InitiatorName='):

                return l[l.index('=')+ 1:].strip()

   

    nova中cinder API封装实现：

    # nova\volume\cinder.py:API.initialize_connection():

    def initialize_connection(self, context, volume, connector):

        return cinderclient(context).\

                 volumes.initialize_connection(volume['id'], connector)

   

    调用的是cinder中的initialize_connection，iscsi driver的实现如下：

    # cinder\volume\iscsi.py:LioAdm.initialize_connection()

    def initialize_connection(self, volume, connector):

        volume_iqn = volume['provider_location'].split(' ')[1]

 

        (auth_method, auth_user, auth_pass)= \

            volume['provider_auth'].split(' ', 3)

 

        # Add initiator iqns to target ACL

        try:

            self._execute('rtstool','add-initiator',

                          volume_iqn,

                          auth_user,

                          auth_pass,

                          connector['initiator'],

                          run_as_root=True)

        except exception.ProcessExecutionErroras e:

            LOG.error(_("Failed to add initiator iqn %s totarget")%

                      connector['initiator'])

            raise exception.ISCSITargetAttachFailed(volume_id=volume['id'])

   

    #nova\virt\libvirt\driver.py:LibvirtDriver.attach_volume()

    @exception.wrap_exception()

    def attach_volume(self, connection_info, instance_name, mountpoint):

        virt_dom = self._lookup_by_name(instance_name)

        mount_device = mountpoint.rpartition("/")[2]

        # 可能需要改动，下面会分析这个方法

        conf = self.volume_driver_method('connect_volume',

                                        connection_info,

                                        mount_device)

 

        if FLAGS.libvirt_type== 'lxc':

            self._attach_lxc_volume(conf.to_xml(), virt_dom, instance_name)

        else:

            try:

                # 挂载到虚拟机上

                virt_dom.attachDevice(conf.to_xml())

            except Exception, ex:

                if isinstance(ex, libvirt.libvirtError):

                    errcode = ex.get_error_code()

                    if errcode== libvirt.VIR_ERR_OPERATION_FAILED:

                        self.volume_driver_method('disconnect_volume',

                                                 connection_info,

                                                 mount_device)

                        raise exception.DeviceIsBusy(device=mount_device)

 

                with excutils.save_and_reraise_exception():

                    self.volume_driver_method('disconnect_volume',

                                              connection_info,

                                              mount_device)

 

        # TODO(danms) once libvirt has support forLXC hotplug,

        # replace this re-define with use of the

        # VIR_DOMAIN_AFFECT_LIVE &VIR_DOMAIN_AFFECT_CONFIG flags with

        # attachDevice()

        # 重新define一下，以间接实现持久化的挂载

        domxml = virt_dom.XMLDesc(libvirt.VIR_DOMAIN_XML_SECURE)

        self._conn.defineXML(domxml)

       

    #nova\virt\libvirt\driver.py:LibvirtDriver.volume_driver_method()

    def volume_driver_method(self, method_name, connection_info,

                             *args,**kwargs):

        driver_type = connection_info.get('driver_volume_type')

        if not driver_type in self.volume_drivers:

            raise exception.VolumeDriverNotFound(driver_type=driver_type)

        driver = self.volume_drivers[driver_type]

        method = getattr(driver, method_name)

        return method(connection_info,*args,**kwargs)

    def __init__():

        ……

        self.volume_drivers= {}

        for driver_strin FLAGS.libvirt_volume_drivers:

            driver_type, _sep, driver= driver_str.partition('=')

            driver_class = importutils.import_class(driver)

            self.volume_drivers[driver_type]= driver_class(self)

    volume_drivers是由配置项libvirt_volume_drivers决定的，默认配置是：

    cfg.ListOpt('libvirt_volume_drivers',

                default=[

                  'iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver',

                  'local=nova.virt.libvirt.volume.LibvirtVolumeDriver',

                  'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver',

                  'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver',

                  'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver'

                  ],

                help='Libvirt handlers for remote volumes.'),

    云硬盘可以使用已有的iscsi driver，也可以参考iscsi实现自己的driver，iscsi driver的内容为：

    # nova\virt\libvirt\volume.py:LibvirtISCSIVolumeDriver:

    class LibvirtISCSIVolumeDriver(LibvirtVolumeDriver):

        """Driver to attachNetwork volumes to libvirt."""

 

        def _run_iscsiadm(self, iscsi_properties, iscsi_command,**kwargs):

            check_exit_code = kwargs.pop('check_exit_code',0)

            (out, err)= utils.execute('iscsiadm','-m','node','-T',

                                      iscsi_properties['target_iqn'],

                                       '-p', iscsi_properties['target_portal'],

                                       *iscsi_command, run_as_root=True,

                                       check_exit_code=check_exit_code)

            LOG.debug("iscsiadm %s:stdout=%s stderr=%s"%

                      (iscsi_command, out, err))

            return (out, err)

 

        def _iscsiadm_update(self, iscsi_properties, property_key, property_value,

                             **kwargs):

            iscsi_command =('--op','update','-n', property_key,

                             '-v', property_value)

            return self._run_iscsiadm(iscsi_properties, iscsi_command, **kwargs)

 

        @utils.synchronized('connect_volume')

        def connect_volume(self, connection_info, mount_device):

            """Attach the volumeto instance_name"""

            iscsi_properties = connection_info['data']

            # NOTE(vish): If we are on the same host asnova volume, the

            #            discovery makes the target so we don't need to

            #            run --op new. Therefore, we check to see if the

            #            target exists, and if we get 255 (Not Found), then

            #            we run --op new. This will also happen if another

            #            volume is using the same target.

            try:

                self._run_iscsiadm(iscsi_properties,())

            except exception.ProcessExecutionErroras exc:

                # iscsiadm returns 21 for "No recordsfound" after version 2.0-871

                if exc.exit_codein [21,255]:

                    self._run_iscsiadm(iscsi_properties,('--op','new'))

                else:

                    raise

 

            if iscsi_properties.get('auth_method'):

                self._iscsiadm_update(iscsi_properties,

                                      "node.session.auth.authmethod",

                                     iscsi_properties['auth_method'])

                self._iscsiadm_update(iscsi_properties,

                                      "node.session.auth.username",

                                     iscsi_properties['auth_username'])

                self._iscsiadm_update(iscsi_properties,

                                      "node.session.auth.password",

                                     iscsi_properties['auth_password'])

 

            # NOTE(vish): If we have another lun on thesame target, we may

            #            have a duplicate login

            self._run_iscsiadm(iscsi_properties,("--login",),

                               check_exit_code=[0,255])

 

            self._iscsiadm_update(iscsi_properties,"node.startup","automatic")

 

            host_device =("/dev/disk/by-path/ip-%s-iscsi-%s-lun-%s"%

                            (iscsi_properties['target_portal'],

                             iscsi_properties['target_iqn'],

                             iscsi_properties.get('target_lun',0)))

 

            # The /dev/disk/by-path/... node is notalways present immediately

            # TODO(justinsb): This retry-with-delay is apattern, move to utils?

            tries =0

            while not os.path.exists(host_device):

                if tries>= FLAGS.num_iscsi_scan_tries:

                    raise exception.NovaException(_("iSCSI device not found at %s")

                                                 %(host_device))

 

                LOG.warn(_("ISCSI volume not yet found at:%(mount_device)s. "

                           "Will rescan & retry.  Try number: %(tries)s")%

                         locals())

 

                # The rescan isn't documented as beingnecessary(?), but it helps

                self._run_iscsiadm(iscsi_properties,("--rescan",))

 

                tries = tries+ 1

                if not os.path.exists(host_device):

                    time.sleep(tries** 2)

 

            if tries!= 0:

                LOG.debug(_("Found iSCSI node %(mount_device)s "

                            "(after %(tries)s rescans)")%

                          locals())

 

            connection_info['data']['device_path']= host_device

            sup = super(LibvirtISCSIVolumeDriver, self)

            return sup.connect_volume(connection_info, mount_device)

 

        @utils.synchronized('connect_volume')

        def disconnect_volume(self, connection_info, mount_device):

            """Detach the volumefrom instance_name"""

            sup = super(LibvirtISCSIVolumeDriver, self)

            sup.disconnect_volume(connection_info, mount_device)

            iscsi_properties = connection_info['data']

            # NOTE(vish): Only disconnect from the targetif no luns from the

            #            target are in use.

            device_prefix =("/dev/disk/by-path/ip-%s-iscsi-%s-lun-"%

                             (iscsi_properties['target_portal'],

                              iscsi_properties['target_iqn']))

            devices = self.connection.get_all_block_devices()

            devices = [dev for dev in devices if dev.startswith(device_prefix)]

            if not devices:

                self._iscsiadm_update(iscsi_properties,"node.startup","manual",

                                     check_exit_code=[0,255])

                self._run_iscsiadm(iscsi_properties,("--logout",),

                                  check_exit_code=[0,255])

                self._run_iscsiadm(iscsi_properties,('--op','delete'),

                                  check_exit_code=[0,21,255])

也即主要实现了卷挂载到宿主机和从宿主机卸载两个方法。

 

 

2.3    相关代码源文件

nova\volume\cinder.py源文件（云硬盘需要实现的方法或者要封装的API都在这里面）：   https://github.com/openstack/nova/blob/stable/folsom/nova/volume/cinder.py       

nova\virt\libvirt\volume.py源文件（云硬盘需要实现的driver可以参考这个文件）：   https://github.com/openstack/nova/blob/stable/folsom/nova/virt/libvirt/volume.py

# 默认的driver映射关系，可以看出iscsi卷使用的是LibvirtISCSIVolumeDriver

cfg.ListOpt('libvirt_volume_drivers',

    default=[

  'iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver',

            'local=nova.virt.libvirt.volume.LibvirtVolumeDriver',                 'fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver',

            'rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver',

            'sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver'

            ],

     help='Libvirt handlers for remote volumes.'),

   

cinder处理各种API请求的抽象类源文件：    https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py

上述抽象类会调用不同的driver去执行实际的动作，完成API的请求，其中iSCSI driver源文件为：   

    # 默认的volume driver是cinder.volume.drivers.lvm.LVMISCSIDriver

    cfg.StrOpt('volume_driver',

               default='cinder.volume.drivers.lvm.LVMISCSIDriver',

               help='Driver to use for volume creation'),

    ]    https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/lvm.py#L304

它继承了LVMVolumeDriver, driver.ISCSIDriver两个类，其中后一个类所在的源文件为：   https://github.com/openstack/cinder/blob/master/cinder/volume/driver.py#L199   https://github.com/openstack/cinder/blob/master/cinder/volume/driver.py#L339这里的self.tgtadm是在   https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/lvm.py#L321这里初始化的，调用的是   https://github.com/openstack/cinder/blob/master/cinder/volume/iscsi.py#L460这里的方法。

iscsi_helper默认使用的是tgtadm：

    cfg.StrOpt('iscsi_helper',

               default='tgtadm',

               help='iscsi target user-land tool to use'),

3.   需要新增的API

1)        扩容云硬盘的API（或者直接调用云硬盘已有的API，但是推荐nova新增一个，这样云硬盘就不必对外暴露任何API了，都可以经过nova来转发处理。）

   

4.   需要注意的问题

1)        之前云硬盘agent实现的一下错误恢复、异常处理逻辑需要在nova里面实现

2)        挂载点在云主机内外看到的不一致问题（因为nova挂载动作是异步的，所以返回给用户的是libvirt看到的挂载点，不是实际的虚拟机内部的挂载点，目前考虑通过查询卷信息接口返回最终的挂载点）

3)        用户及认证问题（之前云硬盘应该用的是管理平台的用户认证逻辑，如果改为使用nova接口，需要使用keystone的用户认证，不知道可否在管理平台那一层转换一下）

   

总的来说云硬盘所需要做的改动应该不大，工作重点在于封装已有的API，提供client即可（参考https://github.com/openstack/nova/blob/stable/folsom/nova/volume/cinder.py），另外driver（参考https://github.com/openstack/nova/blob/stable/folsom/nova/virt/libvirt/volume.py）里面要实现扩容逻辑，应该可以重用agent中现有的代码。