I tested Xen v4.1 + Linux kernel 3.2.0-68 in Ubuntu 12.04. Suspend is okay, but resumes did not.
My labmate Nilton found that:
Xen 4.x has another issue that prevents it from resuming from S3. This threadhttp://lists.xenproject.org/archives/html/xen-devel/2013-06/msg02947.html suggests Xen 4.2.3 with a set of patches on Linux 3.8.8 to fix that.
Our final solution is use KVM + Xen. We run KVM as the hypervisor. The reason of using KVM is that Ubuntu has better support for Suspend to RAM.
We then start a KVM guest and installed Xen 3.4 on it. It didn't work at first. Xen panics at probing a MSR feature of the CPU. We commented out a line in Xen to disable the MSR probe! Then it works: (1) Xen can boot as a KVM guest, (2) we start new jettison VMs on Xen, (3) VM suspend/resume works fine.
Below is the panic message before commenting out the MSR probe line:
(XEN) Xen version 3.4.0 (root@) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Wed Sep 3 09:55:51 EDT 2014
(XEN) Latest ChangeSet: Wed Feb 24 10:59:37 2010 +0000 29409:a705ab9e2760
(XEN) Command line: com1=38400,8n1 console=com1 nolapic
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 000000000009dc00 (usable)
(XEN) 000000000009dc00 - 00000000000a0000 (reserved)
(XEN) 00000000000f0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000dfffd000 (usable)
(XEN) 00000000dfffd000 - 00000000e0000000 (reserved)
(XEN) 00000000feffc000 - 00000000ff000000 (reserved)
(XEN) 00000000fffc0000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000001f99500000 (usable)
(XEN) System RAM: 128916MB (132010600kB)
(XEN) ACPI: RSDP 000FDA30, 0014 (r0 BOCHS )
(XEN) ACPI: RSDT DFFFD210, 0034 (r1 BOCHS BXPCRSDT 1 BXPC 1)
(XEN) ACPI: FACP DFFFFC10, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1)
(XEN) ACPI: DSDT DFFFD630, 2589 (r1 BXPC BXDSDT 1 INTL 20100528)
(XEN) ACPI: FACS DFFFFBC0, 0040
(XEN) ACPI: SSDT DFFFD3A0, 0283 (r1 BOCHS BXPCSSDT 1 BXPC 1)
(XEN) ACPI: APIC DFFFD290, 009A (r1 BOCHS BXPCAPIC 1 BXPC 1)
(XEN) ACPI: HPET DFFFD250, 0038 (r1 BOCHS BXPCHPET 1 BXPC 1)
(XEN) Domain heap initialised
(XEN) Processor #0 6:2 APIC version 20
(XEN) Processor #1 6:2 APIC version 20
(XEN) Processor #2 6:2 APIC version 20
(XEN) Processor #3 6:2 APIC version 20
(XEN) Processor #4 6:2 APIC version 20
(XEN) Processor #5 6:2 APIC version 20
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.051 MHz processor.
(XEN) Xen BUG at vmcs.c:87
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff828c8019f42a>] vmx_cpu_up+0x1aa/0x760
(XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor
(XEN) rax: 00000000fffffdff rbx: 0000000000181000 rcx: 0000000000000483
(XEN) rdx: 0000000000000200 rsi: 0000000000000200 rdi: 0000000000000001
(XEN) rbp: ffff828c80277f28 rsp: ffff828c80277d68 r8: 00000000b2998688
(XEN) r9: 000000000000003f r10: 0000000011e57ed0 r11: 000000008028fea0
(XEN) r12: ffff828c801f8c88 r13: 000000000000000c r14: 0000000000000005
(XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000020b0
(XEN) cr3: 00000000df87c000 cr2: 0000000000000000
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008
(XEN) Xen stack trace from rsp=ffff828c80277d68:
(XEN) 0000000000000010 0000000000000001 0000000000007ff1 0000000000000000
(XEN) ffff8300ffe00000 ffff8300df881ff0 000ffffffffff000 0000000000000800
(XEN) ffff828c80277f28 ffff828c801f8c88 000000000000000c 0000000000100000
(XEN) ffff83000008bf60 ffff828c801a3191 0000062300000800 0000000000000000
(XEN) ffff828c801f8c80 ffff828c8017e6cf 8080202100000800 00000623078bfbfd
(XEN) ffff828c80277f28 ffff828c801f8c80 ffff828c801f8c88 ffff828c8017e392
(XEN) ffff828c80277f28 ffff831f994f6000 0000000000000000 ffff828c8021f3d1
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 00000000002ba3c0
(XEN) ffffff8c8027c000 0008bf6000000000 000000000008bf60 0000000000000000
(XEN) ffff83000008bfc0 00000000074a99c0 0000000000000000 ffffffff00000000
(XEN) 0000000800000000 000000010000006e 0000000000000003 00000000000002f8
(XEN) 0000000000000000 0000000000000000 0000000000067e9c 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 ffff828c801000b5
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<ffff828c8019f42a>] vmx_cpu_up+0x1aa/0x760
(XEN) [<ffff828c801a3191>] start_vmx+0xd1/0x1c0
(XEN) [<ffff828c8017e6cf>] init_intel+0x15f/0x310
(XEN) [<ffff828c8017e392>] identify_cpu+0xd2/0x250
(XEN) [<ffff828c8021f3d1>] __start_xen+0x16a1/0x27c0
(XEN) [<ffff828c801000b5>] __high_start+0xa1/0xa3
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at vmcs.c:87
(XEN) ****************************************