Running Nested VMs

Running Nested VMs

版本 36  

创建于: 2008-11-30 下午6:17 作者  jmattson  - 最后修改:  2011-9-27 上午7:38 作者  jmattson

http://communities.vmware.com/docs/DOC-8970

Terminology

The phrase "running nested VMs" refers to running a VM inside another VM. The outer guest is the VM that runs on physical hardware. The inner guest (or nested guest) is the VM that runs within another VM. The host hypervisor is the hypervisor that runs on the physical hardware. The guest hypervisor is the hypervisor that runs within a VM. To avoid confusion, we will not consider multiple levels of nesting here.

Running Guest Hypervisors with Virtualized HV

Most hypervisors require hardware-assisted virtualization (HV). VMware products require hardware-assisted virtualization for 64-bit guests on Intel hardware. When running as a guest hypervisor, VMware products also require hardware-assisted virtualization for 64-bit guests on AMD hardware.  The hardware-assisted virtualization features of the physical CPU are not typically available in a VM, because most hypervisors (from VMware or others) do not virtualize HV. However, Workstation 8, Fusion 4, and ESXi 5.0 offer virtualized HV, so that you can run guest hypervisors which require hardware-assisted virtualization.

 

With virtualized HV enabled for the outer guest, you should be able to run any guest hypervisor that requires hardware-assisted virtualization. In particular, this means that you will be able to run 64-bit nested guests under VMware guest hypervisors.

 

Workstation 8

Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI. To enable virtualized HV, select VM->Settings and navigate to the processor settings screen. Check the box next to "Virtualize Intel VT-x/EPT or AMD-V/RVI."  You will see a warning that virtualized HV will make this VM incompatible with other VMware products. In particular, if this VM is moved to an ESXi 5.0 host, virtualized HV will not be available without additional tweaking.

Fusion 4

Virtualized HV is available for virtual hardware version 8 VMs on hosts that support Intel VT-x and EPT or AMD-V and RVI, but it cannot be selected through the user interface. To enable virtualized HV, edit the outer guest's configuration file by hand and add the following line:

vhv.enable = TRUE

ESXi 5.0

On ESXi 5.0, virtualized HV is prohibited by default. This feature is used internally within VMware for testing purposes, but it is not recommended for production systems. It is available on hosts that support Intel VT-x or AMD-V, but it is not recommended for systems without second level address translation (EPT or RVI), because of its poor performance without SLAT.

 

To allow the use of this feature, the ESXi administrator must add the following configuration option to the/etc/vmware/config file on the physical host:

vhv.allow = TRUE

Once allowed by the ESXi administrator, virtualized HV will be enabled by default for all hardware version 8 VMs with guest OS type "ESX Server 4" and "ESX Server 5." To enable virtualized HV for other guests, add the following lines to the outer guest's configuration file:

cpuid.1.ecx="----:----:----:----:----:----:--h-:----"
cpuid.80000001.ecx.amd="----:----:----:----:----:----:----:-h--"
cpuid.8000000a.eax.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.ebx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"
cpuid.8000000a.edx.amd="hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh:hhhh"

Running Guest Hypervisors without Virtualized HV

VMware products prior to ESXi 5.0, Workstation 8 or Fusion 4 do not virtualize the hardware-assisted virtualization capabilities of the physical processor (or 64-bit segment limit checks on AMD processors). If the host hypervisor is an older VMware product, or if your host does not support EPT or RVI (but it does support Intel VT-x or AMD-V) you can still run nested guests without virtualized HV. However, you will only be able to run 32-bit nested guests using binary translation under a VMware guest hypervisor. You can also run 32-bit nested guests using the binary translation features of Windows Virtual PC or VPC2007.

 

Even in this configuration, running nested VMs requires host support for hardware-assisted virtualization.  Furthermore, the outer guest must be configured to use hardware-assisted virtualization.  For instructions on enabling hardware-assisted virtualization for the outer guest, see The specified document was not found..

Special Considerations

Networking in Nested Guests

For networking to work in a nested guest, the network interfaces of the outer guest must be put into promiscuous mode. For details, see KB 287 (Linux hosts) or KB 1004099 (ESX hosts). No special configuration is required for Windows hosts.

 

EVC Clusters

Virtualized HV may not work on hosts that are part of an EVC cluster, due to the feature masking that is performed to make all of the CPUs in the cluster look alike.

 

VMware Tools

Beginning with ESXi 5.0, Workstation 8 and Fusion 4, VMware hypervisors implement a handshake between the host hypervisor and the guest hypervisor which allows you to run VMware Tools in both the inner guest and the outer guest. Note that both hypervisors must be of this recent vintage for the handshake to work.

 

If either hypervisor is older than this, you will only be able to run VMware Tools in the inner guest. To ensure that VMware Tools work properly, you must either set the outer guest OS type to "ESX Server 4" or "ESX Server 5," or you must add the following line to the outer guest's configuration file:

monitor_control.restrict_backdoor = TRUE

Note that this setting is not available when your host hypervisor is ESX 3.0. You cannot use VMware Tools in the inner guest if your host hypervisor is ESX 3.0.

Foreign Guest Hypervisors

Windows Virtual PC (XP mode)

The files required to run an XP mode VM under Windows 7 are available from http://www.microsoft.com/windows/virtual-pc/download.aspx.  To use Windows Virtual PC with binary translation, all three downloads are needed. Without the binary translator, Windows Virtual PC requires hardware-assisted virtualization.

 

Windows Virtual PC has a bug related to power-saving states when running with Intel VT-x. If you try to sleep or hibernate the outer Windows VM while a nested Windows Virtual PC VM is running, you may run into a variety of issues when the outer guest is awakened. To solve this problem, you will have to install the hotfix from http://support.microsoft.com/?kbid=977632. Even if the description does not sound like it is applicable, this hotfix should help with any sleep/hibernate issues.

Hyper-V

Hyper-V requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8 or Fusion 4. Hyper-V performs relatively poorly as a guest hypervisor under ESXi 5.0, but it performs reasonably well under Workstation 8 or Fusion 4.

 

In addition to the configuration options given above, Hyper-V requires the following additional configuration option for the outer guest:

hypervisor.cpuid.v0 = FALSE

This option hides the fact that Hyper-V is running inside a VM.

 

Without this option, launching a nested VM under Hyper-V R2 will fail with the following error:

Failed to create partition: Unspecified error (0x80004005) 

Without this option, Hyper-V R3 will refuse to install, claiming:

Hyper-V cannot be installed: A hypervisor is already running.

Xen

Xen requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8 or Fusion 4. The version of Xen provided with most Linux distributions performs relatively poorly as a guest hypervisor. Citrix XenServer 5.6 performs reasonably well.

KVM

KVM requires hardware-assisted virtualization, so it can only be run under ESXi 5.0, Workstation 8 or Fusion 4. KVM performs relatively poorly as a guest hypervisor on Intel CPUs using virtualized Intel VT-x. Performance is improved with Linux kernel versions 3.0 and greater.

 

VMware as a Guest Hypervisor Under Foreign Hypervisors

If we detect that a VMware product is running under a foreign hypervisor, we will refuse to power on a nested guest. To bypass this constraint, add the following option to your nested guest's configuration file:

vmx.allowNested = TRUE

 

VMware Workstation under Hyper-V

You can run the VMware binary translator (for 32-bit guests) under Hyper-V if your host has SLAT support.  Without SLAT support, if you attempt to run a nested VM under VMware Workstation, Hyper-V will synthesize a machine check and BSOD the host.

 

The Workstation installer won't allow you to install Workstation while the Hyper-V role is active, so you will have to disable Hyper-V to do the installation.  See http://blogs.msdn.com/b/virtual_pc_guy/archive/2008/04/14/creating-a-no-hypervisor-boot-entry.aspx for instructions on how to boot Windows 2008 with Hyper-V disabled, without actually having to remove the Hyper-V role.