Openstack liberty 云主机迁移源码分析之静态迁移2

接上一篇Openstack liberty 云主机迁移源码分析之静态迁移1，上篇文中主要介绍了nova-api及nova-conductor的处理过程，本文将重点分析在云主机迁移过程中nova-compute所做的工作，可以分为如下三个部分：

• prepare准备阶段
• execute执行阶段
• complete完成阶段

下面一起来看看具体的内容：

prepare准备阶段

nova-compute从消息队列拿到prep_resize请求后，将由下述方法处理该请求：

#/nova/compute/manager.py/ComputeManager.prep_resize
def prep_resize(self, context, image, instance, instance_type,
                    reservations, request_spec, 
                    filter_properties, node,
                    clean_shutdown):
    """Initiates the process of moving a running instance to 
    another host.Possibly changes the RAM and disk size in the 
    process.

    输入参数，来自`nova-conductor`，如下：
    image 云主机所使用的镜像信息
    instance InstanceV2对象，包含云主机详细信息
    instance_type， Flavor对象，云主机所使用的配置模板
    reservations = []  资源保留配额
    request_spec 请求参数，包含：镜像信息，云主机信息，配置模板
    filter_properties 过滤属性
    node 目标节点名， 这里是`devstack`
    """

    #如果没有指定指定node，则通过hypervisor获取（这里是
    #LibvirtDriver）
    if node is None:
        node = self.driver.get_available_nodes(refresh=True)[0]
        LOG.debug("No node specified, defaulting to %s", node,
                      instance=instance)

    """NOTE(melwitt): Remove this in version 5.0 of the RPC API
    Code downstream may expect extra_specs to be populated 
    since it is receiving an object, so lookup the flavor to 
    ensure this.
    """
    #如果instance_type不是合法的Flavor对象，则从nova.instance_types
    #表中获取配置模板信息
    if not isinstance(instance_type, objects.Flavor):
        instance_type = objects.Flavor.get_by_id(context,
                                       instance_type['id'])
    #从保留配额生成配额对象Quotas
    quotas = objects.Quotas.from_reservations(context,
                                                  reservations,
                                                  instance=instance)

    #异常上下文：迁移发生异常时回滚配额及云主机状态
    with self._error_out_instance_on_exception(context, instance,
                                                   quotas=quotas):
        #发送`compute.instance.exists`通知给ceilometer，
        #通知包含：云主机的详细配置信息；默认审计周期为（month）,  
        #current_period=True,表示添加该通知到当前统计周期                                      
        compute_utils.notify_usage_exists(self.notifier, 
                                            context, instance,
                                           current_period=True)
        #发送`compute.instance.resize.prep.start`通知ceilometer，
        #通知包含：云主机详细信息
        self._notify_about_instance_usage(
                    context, instance, "resize.prep.start")

        try:
            #调用_prep_resize执行后续的迁移操作，下文具体分析
            self._prep_resize(context, image, instance,
                                  instance_type, quotas,
                                  request_spec, 
                                  filter_properties,
                                  node, clean_shutdown)
        # NOTE(dgenin): This is thrown in LibvirtDriver when the
        #               instance to be migrated is backed by LVM.
        #               Remove when LVM migration is implemented.
        #如果nova使用lvm作为后端存储，从镜像启动的云主机将不支持迁移
        except exception.MigrationPreCheckError:
            raise
        except Exception:
            # try to re-schedule the resize elsewhere:
            #获取具体的异常信息，如：UnableToMigrateToSelf
            exc_info = sys.exc_info()
            """重新调度：如果包含retry则执行重新调度，实际上就是通过`
            `nova/conductor/rpcapi.py/ComputeTaskAPI`重新发起
            迁移请求，再次进入云主机迁移源码分析之静态迁移1的`nova-
            conductor`部分，与前一次不同的是，重试请求包含了前一次请求
            的异常信息并且在选择目标主机的时候会排除前一次已选择的目标主
            机
            """
            self._reschedule_resize_or_reraise(context, image,
                        instance,
                        exc_info, instance_type, quotas, 
                        request_spec,
                        filter_properties)
        finally:
            #模板信息：名称及id
            extra_usage_info = dict(
                        new_instance_type=instance_type.name,
                        new_instance_type_id=instance_type.id)
            #发送`compute.instance.resize.prep.end`通知给
            #ceilometer，通知包含：云主机详细信息及配置模板名和id
            self._notify_about_instance_usage(
                    context, instance, "resize.prep.end",
                    extra_usage_info=extra_usage_info)

---------------------------------------------------------------
#接上文：
def _prep_resize(self, context, image, instance, instance_type,
            quotas, request_spec, filter_properties, node,
            clean_shutdown=True):

    if not filter_properties:
        filter_properties = {}

    if not instance.host:
        self._set_instance_obj_error_state(context, instance)
        msg = _('Instance has no source host')
        raise exception.MigrationError(reason=msg)

    #检查源主机是否和目标主机是否是同一个
    same_host = instance.host == self.host

    # if the flavor IDs match, it's migrate; otherwise resize
    #在同一个主机上迁移！！！，需要检查hypervisor是否支持同一主机上的迁移
    if same_host and instance_type.id == 
                                instance['instance_type_id']:
        # check driver whether support migrate to same host
        #libvirt默认不支持同一主机上的迁移，这里会抛异常
        if not self.driver.
                capabilities['supports_migrate_to_same_host']:
            raise exception.UnableToMigrateToSelf(
                    instance_id=instance.uuid, host=self.host)

    # NOTE(danms): Stash the new instance_type to avoid having 
    #to look it up in the database later
    #添加'new_flavor'属性到Instance,包含配置模板instance_type信息
    instance.set_flavor(instance_type, 'new')

    # NOTE(mriedem): Stash the old vm_state so we can set the
    # resized/reverted instance back to the same state later.
    #保留当前的云主机状态，用于回滚或者迁移完成后设置新主机状态
    vm_state = instance.vm_state
    LOG.debug('Stashing vm_state: %s', vm_state, instance=instance)
    instance.system_metadata['old_vm_state'] = vm_state
    instance.save()

    #创建节点资源跟踪器ResourceTracker
    limits = filter_properties.get('limits', {})
    rt = self._get_resource_tracker(node)
    #保存迁移上下文MigrationContext到instance，并添加一条记录到
    #nova.migrations数据表中
    #根据配置模板（instance_type）校验目标主机上资源情况（日志文件中打印
    #Attemping Claim之类的日志，如果资源不足会抛异常）
    #根据配置模板（instance_type）在目标主机上保留迁移所需的资源
    with rt.resize_claim(context, instance, instance_type,
                           image_meta=image, limits=limits) as claim:
        LOG.info(_LI('Migrating'), context=context, 
                                            instance=instance)
        #通过异步rpc发送`resize_instance`请求给`nova-compute`
        #下文具体分析                                 
        self.compute_rpcapi.resize_instance(
                    context, instance, claim.migration, image,
                    instance_type, quotas.reservations,
                    clean_shutdown)

小结： 该阶段主要是完成一些前期的条件判断、参数设置、资源校验及保留及发送ceilometer审计通知，为后续的执行阶段做准备

execute执行阶段

从消息队列拿到前述的resize_instance消息后，nova-compute通过下述方法来执行该请求：

#nova/compute/manager.py/ComputeManager.resize_instance
def resize_instance(self, context, instance, image,
                        reservations, migration, instance_type,
                        clean_shutdown):
    """Starts the migration of a running instance to another 
     host. 

    migration 上文的rpc投递过来的迁移参数，包含：迁移所需的详细信息            
    """  
    #基于资源保留生成配额对象Quotas
    quotas = objects.Quotas.from_reservations(context,
                                                  reservations,
                                                  instance=instance)       
    #异常上下文：迁移发生异常时回滚配额及设置云主机状态                                                      
    with self._error_out_instance_on_exception(context, 
                                        instance,
                                         quotas=quotas):
        """ TODO(chaochin) Remove this until v5 RPC API
        Code downstream may expect extra_specs to be 
        populated since it is receiving an object, so lookup 
        """
        #如果没有指定配置模板，从migration参数中提取
        if (not instance_type or
            not isinstance(instance_type, objects.Flavor)):
            instance_type = objects.Flavor.get_by_id(
                    context, migration['new_instance_type_id'])

        #从neutron数据库中获取与云主机关联的所有网卡信息（VIF信息）
        network_info = self.network_api.get_instance_nw_info(context,
                                                   instance)
        #更新迁移状态(nova.migrations数据表)
        migration.status = 'migrating'
        with migration.obj_as_admin():
            migration.save()

        #更新云主机状态：云主机状态：重建/迁移，任务状态：正在重建或者迁移
        instance.task_state = task_states.RESIZE_MIGRATING
        instance.save(expected_task_state=
                                    task_states.RESIZE_PREP)

        #发送`compute.instance.resize.start`通知给ceilometer，
        #通知包含：云主机详细信息及网卡信息
        self._notify_about_instance_usage(
                context, instance, "resize.start", 
                network_info=network_info)
        #从nova.block_device_mapping数据表中获取云主机关联的块设备映
        #射信息
        bdms = objects.BlockDeviceMappingList.get_by_instance_uuid(
                    context, instance.uuid)
        #将上述的块设备映射信息转换成设备驱动所需要的格式,如：
        """
        {'swap' = None,
         'root_device_name' : u'/dev/vda',
         'ephemerals' = [],
         'block_device_mapping' = []
        }
        block_device_mapping包含，云主机上的卷设备信息列表
        """
        block_device_info = self._get_instance_block_device_info(
                                context, instance, bdms=bdms)

        #从云主机的system_metadata中获取关机超时及重试信息（如果
        #clean_shutdown = True）,否则设置为0，0
        timeout, retry_interval = self._get_power_off_values(context,
                                            instance, clean_shutdown)
        #通过LibvirtDriver执行关机迁移，下文具体分析                                 
        disk_info = self.driver.migrate_disk_and_power_off(
                    context, instance, migration.dest_host,
                    instance_type, network_info,
                    block_device_info,
                    timeout, retry_interval)   
        #通过cinder断开卷设备连接           
        self._terminate_volume_connections(context, instance, bdms)
        #迁移网卡(空操作，在`complete`结束阶段在目的主机上重新配置网卡)
        migration_p = obj_base.obj_to_primitive(migration)
        self.network_api.migrate_instance_start(context,
                                                    instance,
                                                    migration_p)
        #更新迁移状态（nova.migrations数据表）                                            
        migration.status = 'post-migrating'
        with migration.obj_as_admin():
            migration.save()
        #更新云主机的主机信息及状态：云主机状态：重建/迁移，任务状态：正在
        #完成重建或者迁移
        instance.host = migration.dest_compute
        instance.node = migration.dest_node
        instance.task_state = task_states.RESIZE_MIGRATED
        instance.save(expected_task_state=
                                task_states.RESIZE_MIGRATING)
        #通过异步rpc发起`finish_resize`请求，`nova-compute`会处理该
        #请求，下文具体分析
        self.compute_rpcapi.finish_resize(context, 
                        instance,
                        migration, image, disk_info,
                        migration.dest_compute, 
                        reservations=quotas.reservations)
        #发送`compute.instance.resize.end`通知给ceilometer
        #通知包含：云主机详细信息及网卡信息
        self._notify_about_instance_usage(context, instance, 
                        "resize.end",network_info=network_info) 
        #移除所有的pending事件
        self.instance_events.clear_events_for_instance(
                                                    instance)

---------------------------------------------------------------

#接上文：nova/virt/libvirt/driver.py/LibvirtDriver
def migrate_disk_and_power_off(self, context, instance, dest,
                                   flavor, network_info,
                                   block_device_info=None,
                                   timeout=0, retry_interval=0):  
    LOG.debug("Starting migrate_disk_and_power_off",
                   instance=instance)
    #获取外部设备
    ephemerals =  
     driver.block_device_info_get_ephemerals(block_device_info)  
    # get_bdm_ephemeral_disk_size() will return 0 if the new
    # instance's requested block device mapping contain no
    # ephemeral devices. However, we still want to check if
    # the original instance's ephemeral_gb property was set and
    # ensure that the new requested flavor ephemeral size is 
    #greater
    eph_size = (block_device.get_bdm_ephemeral_disk_size(ephemerals) 
                                      or instance.ephemeral_gb) 
    # Checks if the migration needs a disk resize down.
    root_down = flavor.root_gb < instance.root_gb
    ephemeral_down = flavor.ephemeral_gb < eph_size

    #从云主机的xml配置中获取块非volume设备信息，我的例子中为[]
    disk_info_text = self.get_instance_disk_info(
            instance, block_device_info=block_device_info)

    #检查云主机是否从卷启动（如果instance中不包含image_ref镜像属性或者
    #块设备信息中不包含'disk'信息，就是从卷启动的）        
    booted_from_volume = self._is_booted_from_volume(instance,
                                    disk_info_text)
    #如果从镜像启动则根磁盘不支持收缩；外部设备也不支持收缩                                
    if (root_down and not booted_from_volume) or ephemeral_down:
        reason = _("Unable to resize disk down.")
        raise exception.InstanceFaultRollback(
                exception.ResizeError(reason=reason))

    disk_info = jsonutils.loads(disk_info_text)
    # NOTE(dgenin): Migration is not implemented for LVM backed 
    #instances.
    #如果nova使用lvm作为后端存储，从镜像启动的云主机将不支持迁移
    if CONF.libvirt.images_type == 'lvm' and not 
                                        booted_from_volume:
        reason = _("Migration is not supported for LVM backed 
                                                instances")
        raise exception.InstanceFaultRollback(
                exception.MigrationPreCheckError(reason=reason))    

    # copy disks to destination
    # rename instance dir to +_resize at first for using
    # shared storage for instance dir (eg. NFS).
    #获取云主机本地配置路径，
    #如：/opt/stack/data/nova/instances/{uuid}
    inst_base = libvirt_utils.get_instance_path(instance)
    inst_base_resize = inst_base + "_resize"    

    #判断目标主机和源主机是否共享存储
    shared_storage = self._is_storage_shared_with(dest, inst_base)  

    # try to create the directory on the remote compute node
    # if this fails we pass the exception up the stack so we 
    #can catch failures here earlier
    #如果是共享存储，则通过ssh在目标主机上创建云主机根目录，失败则抛异常
    if not shared_storage:
        try:
            self._remotefs.create_dir(dest, inst_base)
        except processutils.ProcessExecutionError as e:
            reason = _("not able to execute ssh command: %s") % e
            raise exception.InstanceFaultRollback(
                    exception.ResizeError(reason=reason))   
    #执行迁移前，通过libvirt关闭云主机
    self.power_off(instance, timeout, retry_interval)  

    #从块设备信息字典中获取卷设备映射字典
    block_device_mapping = driver.block_device_info_get_mapping(
            block_device_info)
    #迁移前，通过特定类型的卷驱动卸载卷设备。对于
    #rbd（LibvirtNetVolumeDriver），什么都没有做；
    #对于iscsi（LibvirtISCSIVolumeDriver），做了两个工作：
    #1. echo '1' > /sys/block/{dev_name}/device/delete
    #2. 通过iscsiadm工具删除相关的端点信息
    for vol in block_device_mapping:
        connection_info = vol['connection_info']
        disk_dev = vol['mount_device'].rpartition("/")[2]
        self._disconnect_volume(connection_info, disk_dev)

    try:
        #重命名云主机配置目录
        utils.execute('mv', inst_base, inst_base_resize)
        """ if we are migrating the instance with shared 
        storage then create the directory.  If it is a remote 
        has already been created
        """
        #创建新的云主机配置目录
        if shared_storage:
            dest = None
            utils.execute('mkdir', '-p', inst_base)

        #创建一个作业跟踪器（添加/删除），用于后面的磁盘拷贝
        on_execute = lambda process: \
                self.job_tracker.add_job(instance, process.pid)
        on_completion = lambda process: \
                self.job_tracker.remove_job(instance, process.pid)
        #获取云主机配置模板        
        active_flavor = instance.get_flavor()

        #迁移非volume设备
        for info in disk_info:
            # assume inst_base == dirname(info['path'])
            img_path = info['path']
            fname = os.path.basename(img_path)
            from_path = os.path.join(inst_base_resize, fname)

            """ To properly resize the swap partition, it must 
            be re-created with the proper size.  This is 
            acceptable because when an OS is shut down, the 
            contents of the swap space are just garbage, the OS 
            doesn't bother about what is in it.
            We will not copy over the swap disk here, and rely 
            on finish_migration/_create_image to re-create it 
            for us.
            """
            if not (fname == 'disk.swap' and
                    active_flavor.get('swap', 0) != 
                        flavor.get('swap', 0)):
                compression = info['type'] not in 
                                    NO_COMPRESSION_TYPES
                libvirt_utils.copy_image(from_path, img_path, 
                                        host=dest,
                                        on_execute=on_execute,
                                   on_completion=on_completion,
                                     compression=compression)
        """ Ensure disk.info is written to the new path to 
        avoid disks being reinspected and potentially 
        changing format.
        """
        #拷贝disk_info到目的主机（如果有的话），我是采用ceph 作为nova
        #后端存储，在根目录上只有console.log和libvirt.xml两个文件
        src_disk_info_path = os.path.join(inst_base_resize, 
                                                   'disk.info')
        if os.path.exists(src_disk_info_path):
            dst_disk_info_path = os.path.join(inst_base, 
                                                   'disk.info')
            libvirt_utils.copy_image(src_disk_info_path,
                                         dst_disk_info_path,
                                         host=dest, 
                                         on_execute=on_execute,
                                on_completion=on_completion)

    except Exception:
        #发生异常回滚上述的操作，并上抛异常
        with excutils.save_and_reraise_exception():
            self._cleanup_remote_migration(dest, inst_base,
                                              inst_base_resize,
                                               shared_storage)

    return disk_info_text

小结：该阶段主要完成non-volume块设备的复制，同时更新云主机状态并发送ceilometer审计通知，最后通过异步rpc发起finish_resize请求，进入下一阶段

complete完成阶段

从消息队列拿到上述的finish_resize消息后，nova-compute通过下述方法来执行该请求：

#nova/compute/manager.py/ComputeManager.finish_resize
def finish_resize(self, context, disk_info, image, instance,
                      reservations, migration):
    """Completes the migration process.
    Sets up the newly transferred disk and turns on the 
    instance at its new host machine.
    """
    #生成配额对象Quotas
    quotas = objects.Quotas.from_reservations(context,
                                                  reservations,
                                                  instance=instance)
    try:
        #完成迁移的结尾操作，下文具体分析
        self._finish_resize(context, instance, migration,
                                disk_info, image)
        #提交资源配额，更新数据库
        quotas.commit()
    except Exception:
        LOG.exception(_LE('Setting instance vm_state to ERROR'),
                          instance=instance)
        #发生异常，回滚配额
        with excutils.save_and_reraise_exception():
            try:
                quotas.rollback()
            except Exception:
                LOG.exception(_LE("Failed to rollback quota" 
                                 "for failed finish_resize"),
                                  instance=instance)
            #设置云主机状态为error（错误）                      
            self._set_instance_obj_error_state(context, instance)

------------------------------------------------------------
#接上文：
def _finish_resize(self, context, instance, migration, disk_info,
                       image):
    #默认是迁移操作
    resize_instance = False
    #从迁移参数字典中提取新旧配置模板id
    old_instance_type_id = migration['old_instance_type_id']
    new_instance_type_id = migration['new_instance_type_id']
    #从云主机实例对象提取配置模板信息
    old_instance_type = instance.get_flavor()
    """ NOTE(mriedem): Get the old_vm_state so we know if we 
    should power on the instance. If old_vm_state is not set we 
    need to default to ACTIVE for backwards compatibility
    """
    #从云主机实例对象的系统元信息中提取迁移前云主机的状态，默认为运行态
    #在`prepare`准备阶段，会设置该值
    old_vm_state = instance.system_metadata.get('old_vm_state',
                                                   vm_states.ACTIVE)
    #添加old_flavor属性,保存旧的配置模板到云主机实例对象中                                               
    instance.set_flavor(old_instance_type, 'old')

    #如果新旧配置模板id不同，则用新模板配置instance
    if old_instance_type_id != new_instance_type_id:
        #从new_flavor属性获取新配置模板，该值在`prepare`准备阶段设置
        instance_type = instance.get_flavor('new')
        self._set_instance_info(instance, instance_type)
        #如果新旧配置中根磁盘、交换分区、外设不同，则认为是resize操作
        for key in ('root_gb', 'swap', 'ephemeral_gb'):
            if old_instance_type[key] != instance_type[key]:
                resize_instance = True
                break
    #应用迁移上下文MigrationContext，上下文在`prepare`准备阶段设置
    #更新instance的numa拓扑信息
    instance.apply_migration_context()

    # NOTE(tr3buchet): setup networks on destination host
    #在目标主机上为新的云主机配置网卡（这是空操作，下面的
    #migrate_instance_finish才执行正在的网卡配置）
    self.network_api.setup_networks_on_host(context, instance,
                                     migration['dest_compute'])
    #迁移网卡，更新数据库（设置'binding:host_id'属性为目标主机）
    migration_p = obj_base.obj_to_primitive(migration)
    self.network_api.migrate_instance_finish(context,
                                                 instance,
                                                 migration_p)
    #获取云主机的网卡信息，发送错误则抛异常
    network_info = self.network_api.get_instance_nw_info(context, instance)

    #更新云主机状态： 云主机状态：重建/迁移，任务状态：完成重建或者迁移
    instance.task_state = task_states.RESIZE_FINISH
    instance.save(expected_task_state=
                            task_states.RESIZE_MIGRATED)

    #发送`compute_instance.finish_resize.start`通知给ceilometer
    #通知包含云主机详细信息及网卡信息
    self._notify_about_instance_usage(
            context, instance, "finish_resize.start",
            network_info=network_info)

    #获取云主机的块设备信息，并更新卷设备的'connection_info'信息
    block_device_info = self._get_instance_block_device_info(
                            context, instance, 
                            refresh_conn_info=True)

    """ NOTE(mriedem): If the original vm_state was STOPPED, we 
    don't automatically power on the instance after it's 
    migrated
    """
    #需要启动云主机
    power_on = old_vm_state != vm_states.STOPPED
    try:
        #在目标主机上配置云主机，并启动云主机（如果需要）
        """这部分的处理代码与之前写的[云主机启动过程源码分析3](http://blog.csdn.net/lzw06061139/article/details/51505514)
        中的最后一部分代码相似，都是调用相同的接口完成磁盘配置，网络配置，
        xml配置等，最后启动云主机；
        """
        self.driver.finish_migration(context, migration, 
                                        instance,
                                         disk_info,
                                         network_info,
                                         image, 
                                         resize_instance,
                                         block_device_info, 
                                         power_on)
    except Exception:
       #异常，还原云主机配置模板
        with excutils.save_and_reraise_exception():
            if old_instance_type_id != new_instance_type_id:
                self._set_instance_info(instance,
                                            old_instance_type)
    #更新迁移状态（nova.migrations数据表）
    migration.status = 'finished'
    with migration.obj_as_admin():
        migration.save()

    #更新云主机状态：任务状态为：None
    instance.vm_state = vm_states.RESIZED
    instance.task_state = None
    instance.launched_at = timeutils.utcnow()
    instance.save(expected_task_state=
                            task_states.RESIZE_FINISH)

    #发送消息给`nova-scheduler`，更新节点上的主机信息
    self._update_scheduler_instance_info(context, instance)

    #发送`compute_instance.finish_resize.end`通知给ceilometer，
    #通知内容包含云主机详细信息及网卡信息
    self._notify_about_instance_usage(
            context, instance, "finish_resize.end",
            network_info=network_info)

小结：该阶段主要是配置云主机：磁盘，网卡，生成xml文件，然后启动云主机。

到这里云主机静态迁移的源码分析就完成了，纵观整个迁移过程：openstack的大部分工作是在处理各种资源配置，一切资源就绪后，其实就是启动一个新的云主机。