技术并艺术着

张华的技术Blog

How to debug QEMU (by quqi99)

**作者:张华 发表于:2016-08-11
版权声明:可以任意转载,转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
( http://blog.csdn.net/quqi99 )**

LOG

1, Capture DEBUG for QEMU & libvirt but only WARN + ERROR for the rest, modify the file /etc/libvirt/libvirtd.conf

log_filters="1:qemu 1:libvirt 3:security 3:event 3:util 3:file"
log_outputs="1:file:/var/log/libvirt/libvirtd.log"

2, To log all libvirt API calls, export (/etc/default/libvirt-bin):

LIBVIRT_DEBUG=1
LIBVIRT_LOG_FILTERS="1:qemu"
LIBVIRT_LOG_OUTPUTS="1:journald 1:file:virsh.log"

3, sudo service libvirt-bin restart
/var/log/libvirt/qemu/instance-00000002.log
/var/log/libvirt/libvirtd.log

4, qemu’s log, add the following
-d guest_errors,unimp -D /tmp/qemu.log

ftrace trace howto

qemu supports multiple trace backends like nop, dtrace, ftrace, log, simple, ust, This is a blog about ftrace howto - http://git.qemu.org/?p=qemu.git;a=commit;h=781e9545dbe54d2f0a530edbf48fd0ac1e77244c

./configure –trace-backend=ftrace
make
sudo echo 1 > /sys/kernel/debug/tracing/events/kvm/enable
qemu-system-x86_64 -enable-kvm -trace events=/home/events
less /sys/kernel/debug/tracing/trace
qemu-system-x86-23226 [002] d… 116142.685923: kvm_exit: reason IO_INSTRUCTION rip 0xc45b info 700040 0
qemu-system-x86-23226 [002] …. 116142.685924: kvm_pio: pio_write at 0x70 size 1 count 1

Observe any arbitrary QMP events

sudo virsh qemu-monitor-command --hmp trusty 'help'
# https://qemu.weilnetz.de/doc/qemu-doc.html#pcsys_005fmonitor
sudo virsh qemu-monitor-command --hmp trusty 'stop'
sudo virsh qemu-monitor-command --hmp trusty 'c'
sudo virsh qemu-monitor-command --hmp trusty 'info network'
#sudo virsh qemu-monitor-command instance-00000002 --pretty '{"execute":"query-commands"}'
#sudo virsh qemu-monitor-command instance-00000002 --pretty '{"execute":"query-block"}'

# Then we can read related code about 'qemuProcessEventHandler -> processWatchdogEvent'
sudo virsh qemu-monitor-event instance-00000002 --pretty --loop

#nova live-migrate --block-migrate vm1 $DEST-HOST
#sudo virsh migrate –verbose --copy-storage-inc --p2p --live vm1 qemu+ssh://root@dest/system
$ sudo virsh qemu-monitor-event instance-00000002 --pretty --loop
error: internal error: virsh qemu-monitor-event: no domain VSH_OT_DATA option
event POWERDOWN at 1470898098.552573 for domain instance-00000002: <null>
event NIC_RX_FILTER_CHANGED at 1470898099.457593 for domain instance-00000002: {
    "name": "net0",
    "path": "/machine/peripheral/net0/virtio-backend"
}
event SHUTDOWN at 1470898100.598173 for domain instance-00000002: <null>
event STOP at 1470898100.600347 for domain instance-00000002: <null>
event SHUTDOWN at 1470898100.601556 for domain instance-00000002: <null>
event RESUME at 1470898102.503793 for domain instance-00000002: <null>
event NIC_RX_FILTER_CHANGED at 1470898122.381443 for domain instance-00000002: {
    "name": "net0",
    "path": "/machine/peripheral/net0/virtio-backend"
}
**

Monitor

#virtio-balloon stats virtqueue test

hua@node1:/images/kvm$ sudo qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -drive if=virtio,cache=none,format=qcow2,file=./ubuntu14.04_test.qcow2 -device virtio-balloon-pci,id=virtio-balloon0 -S -monitor stdio 
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) qom-set virtio-balloon0 guest-stats-polling-interval 5
(qemu) c
(qemu) savevm
(qemu) quit

hua@node1:/images/kvm$ sudo qemu-system-x86_64 -enable-kvm -m 1024 -cpu host -drive if=virtio,cache=none,format=qcow2,file=./ubuntu14.04_test.qcow2 -device virtio-balloon-pci,id=virtio-balloon0 -S -monitor stdio 
QEMU 2.5.0 monitor - type 'help' for more information
(qemu) loadvm 1
(qemu) c
#https://en.wikibooks.org/wiki/QEMU/Monitor
# Migration
(qemu) migrate tcp:127.0.0.1:1234   #with: -incoming tcp::1234

(qemu) info blockstats
virtio0: rd_bytes=0 wr_bytes=20480 rd_operations=0 wr_operations=5 flush_operations=2 wr_total_time_ns=859822442 rd_total_time_ns=0 flush_total_time_ns=52308038 rd_merged=0 wr_merged=0 idle_time_ns=287986673745

(qemu) gdbserver tcp::12345
Waiting for gdb connection on device 'tcp::12345'

hua@node1:/bak/linux$ gdb ./qemu -q
./qemu: Success.
(gdb) target remote localhost:12345 
wget http://wiki.qemu.org/download/linux-0.2.img.bz2 -O linux-0.2.img.bz2
bunzip2 -d linux-0.2.img.bz2

#create telent monitor
sudo qemu-system-x86_64 -hda /bak/images/linux-0.2.img -qmp tcp:localhost:4444,server --monitor stdio
telnet localhost 4444

#create unix socket monitor
# 或者当用户在qemu虚拟机界面上输入Ctrl+ALT+2将进入qemu monitor; 然后按CTRL+ALT+1将恢复当正常的虚拟机窗口。
sudo qemu-system-x86_64 /bak/images/linux-0.2.img -qmp unix:/tmp/qmp-sock,server --monitor stdio
sudo nc -U /tmp/qmp-sock

#When not using '--monitor stdio'
{ "execute": "qmp_capabilities" }
{"return": {}}
{ "execute": "query-status" }
{"return": {"status": "running", "singlestep": false, "running": true}}

附录 - VM随机死掉

遇到一个问题,客户在升级的时候发现VM随机死掉。

1, libvirtd daemon (vNetServer side) can handle the event VIR_EVENT_HANDLE_HANGUP in the function virEventPollFromNativeEvents

libvirtd (use virDriverLoadModule to interact with qemu driver) -> virNetDaemonRun -> virEventRunDefaultImpl -> virEventPollRunOnce -> virEventPollDispatchHandles (vireventpoll.c) -> virEventPollFromNativeEvents (set VIR_EVENT_HANDLE_HANGUP according to POLLHUP) -> virEventHandleCallback

int virEventPollToNativeEvents(int events){
int ret = 0;

if (events & VIR_EVENT_HANDLE_HANGUP)
ret |= POLLHUP;
return ret;
}

2, qemu process (vNetClient side) uses the following code path to register the event VIR_EVENT_HANDLE_HANGUP in the function qemuMonitorOpenInternal when starting a VM.

qemuHypervisorDriver -> qemuDomainCreateXML[qemu_driver.c] -> qemuProcessStart -> qemuProcessLaunch -> qemuProcessWaitForMonitor -> qemuConnectMonitor[qemu_process.c] -> qemuMonitorOpen -> qemuMonitorOpenInternal -> virEventAddHandle

if ((mon->watch = virEventAddHandle(mon->fd,
                                    VIR_EVENT_HANDLE_HANGUP |
                                    VIR_EVENT_HANDLE_ERROR |
                                    VIR_EVENT_HANDLE_READABLE,
                                    qemuMonitorIO,
                                    mon,
                                    virObjectFreeCallback)) < 0)

The unix socket between libvirt and qemu is ‘-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00029015.monitor’, the event POLLHUP represents vNetClient Side (qemu side) has closed socket. Why qemu side closed the socket, qemu side should have some problem, so the core dump file is needed for further analysis.

3, qemu side - qemu uses the following code path to start qemu monitor:

main(vl.c, text_console_init) -> mon_init_func -> monitor_init(bind char device for monitoring) -> monitor_qmp_event

starting qemu monitor needs a kind of char device, we are using unix socket, so by checking register_types here the char device should be CHARDEV_BACKEND_KIND_SOCKET, then we have the following code path.

register_types(CHARDEV_BACKEND_KIND_SOCKET) -> qmp_chardev_open_socket -> qemu_chr_open_socket_fd(qemu-sockets.c) -> socket_connect -> unix_connect_opts(opts, errp, callback, opaque) -> qemu_chr_socket_connected(it’s callback) -> check_report_connect_error

static void check_report_connect_error(CharDriverState *chr,Error *err){
TCPCharDriver *s = chr->opaque;
if (!s->connect_err_reported) {
error_report(“Unable to connect character device %s: %s”,
chr->label, error_get_pretty(err));
s->connect_err_reported = true;
}
qemu_chr_socket_restart_timer(chr);
}

void error_vprintf(const char *fmt, va_list ap)
{
if (cur_mon && !monitor_cur_is_qmp()) {
monitor_vprintf(cur_mon, fmt, ap);
} else {
vfprintf(stderr, fmt, ap);
}
}

阅读更多
版权声明:本文为博主原创文章,如需转载,请注明出处! https://blog.csdn.net/quqi99/article/details/52182878
文章标签: qemu debug
个人分类: qemu
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

不良信息举报

How to debug QEMU (by quqi99)

最多只允许输入30个字

加入CSDN,享受更精准的内容推荐,与500万程序员共同成长!
关闭
关闭