记录一次OpenStack由于资源缓存引起虚拟机创建失败的分析过程

目前在OpenStack使用中遇见虚拟机创建失败的问题
问题表现为：

1. 虚拟机调度正常
2. 资源充足
3. 资源配额充足，设置没有限制
4. 主机资源调度返回的资源大小正常与实际匹配
5. 磁盘大小校验不通过，最后limit限制为本地磁盘大小

问题现象：

创建一台虚拟机
状态ERROR, 报错：nova/compute/manager.py 1902
原因是资源limit， 校验不通过：
{u'memory_mb': 130669.0, u'disk_gb': 199.0}

但真实disk size 远远大于199GB

问题分析过程

查看日志

 INFO nova.compute.manager [req-ac70ea37-c0df-4909-8322-... - - -] {u'memory_mb': 130669.0, u'disk_gb': 199.0}
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] Attempting claim: memory 2048 MB, disk 60 GB, vcpus 1 CPU
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] Total memory: 130669 MB, used: 23552.00 MB
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] memory limit: 130669.00 MB, free: 107117.00 MB
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] Total disk: 111710 GB, used: 181.00 GB
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] disk limit: 199.00 GB, free: 18.00 GB
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] Total vcpu: 31 VCPU, used: 4.00 VCPU
 INFO nova.compute.claims [req-ac70ea37-c0df-4909-8322-2...- - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] vcpu limit not specified, defaulting to unlimited
 INFO nova.compute.manager [req-ac70ea37-c0df-4909-8322-... - - -] [instance: BE099D73-3273-489C-B468-1772C17A2A74] Took 0.06 seconds to deallocate network for instance.
 INFO nova.compute.resource_tracker [req-e1d70ac4-a379-4.... - -] Total usable vcpus: 31, total allocated vcpus: 4
 INFO nova.compute.resource_tracker [req-e1d70ac4-a379-4.... - -] Final resource view: name=test phys_ram=130669MB used_ram=23552MB phys_disk=111710GB used_disk=181GB total_vcpus=31 used_vcpus=4 pci_stats=[]
 WARNING nova.scheduler.client.report [req-e1d70ac4-a379.... - - -] Unable to refresh my resource provider record
 INFO nova.compute.resource_tracker [req-e1d70ac4-a379-4.... - -] Compute_service record updated for test:test

查看源码： nova/compute/manager.py

    def _build_and_run_instance(self, context, instance, image, injected_files,
            admin_password, requested_networks, security_groups,
            block_device_mapping, node, limits, filter_properties):

        image_name = image.get('name')
        self._notify_about_instance_usage(context, instance, 'create.start',
                extra_usage_info={'image_name': image_name})

        self._check_device_tagging(requested_networks, block_device_mapping)

        try:
            rt = self._get_resource_tracker(node)
            with rt.instance_claim(context, instance, limits):
                # NOTE(russellb) It's important that this validation be done
                # *after* the resource tracker instance claim, as that is where
                # the host is set on the instance.
                self._validate_instance_group_policy(context, instance,
                        filter_properties)
                image_meta = objects.ImageMeta.from_dict(image)
                with self._build_resources(context, instance,
                        requested_networks, security_groups, image_meta,
                        block_device_mapping) as resources:
                    instance.vm_state = vm_states.BUILDING