VM guests fail to restart after host is gracefully rebooted

Issue

  • kvm guests will not start, keep getting

error: cannot send monitor command to '["execute:qmp_capabilities"]' connection reset by peer

  • kvm host was rebooted while guests were still running and now guests will not start

Environment

  • Red Hat Enterprise Linux 6

Resolution

Delete the corresponding save file under /var/lib/libvirt/qemu/save/ for each guest that will not start correctly

Root Cause

Unknown corruption of save file causes qemu to be unable to restore guest state after host is rebooted while guests are still running

Diagnostic Steps

Check /var/log/libvirt/qemu/<guest>.log for output similar to the following:

 

2011-05-19 10:41:50.729: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm [...]
char device redirected to /dev/pts/2
Using CPU model "cpu64-rhel6"
Using CPU model "cpu64-rhel6"
0+0 records in
0+0 records out
0 bytes (0 B) copied, 5.951e-06 s, 0.0 kB/s
0+4295 records in
0+4295 records out
359626274 bytes (360 MB) copied, 7.67145 s, 46.9 MB/s
2011-05-20 12:04:59.581: shutting down
2011-05-24 11:17:07.347: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm [...]
char device redirected to /dev/pts/1
Using CPU model "cpu64-rhel6"
Using CPU model "cpu64-rhel6"
qemu: warning: error while loading state section id 4
load of migration failed
2011-05-24 11:17:09.522: shutting down

 

The "359626274 bytes (360 MB) copied, 7.67145 s, 46.9 MB/s" output is from libvirt saving the guest memory to disk because the host is shutting down while the guest is still running.  On the next boot libvirt will, by default, automatically try to restore the guest back to this saved state.  The process of saving and restoring the state of a VM is very similar to how live migrations of guests take place.  QEMU warns that there is a problem with the saved state, and loading it fails shortly after trying to start guest.