![149be1c0699ed002fd0630f696bef1bd.png](https://i-blog.csdnimg.cn/blog_migrate/8bb966c159c267c599fac0670369efc2.png)
![2a8416b661e42893046ff6c5940e22e1.png](https://i-blog.csdnimg.cn/blog_migrate/c63f7080dec94d36efff65a98c777b76.png)
![a2eef12a9e1d9b12e982141ff008f7d1.png](https://i-blog.csdnimg.cn/blog_migrate/fa556e95a9f82f83df1c37d9bc035397.jpeg)
谢宝友,阿里巴巴系统软件事业部内核软件工程师。他以奇特的经历进入IT行业,在编程一线工作已经有20年时间,其中最近10年时间工作于Linux操作系统内核。在中兴通讯操作系统产品部工作期间,他作为技术总工参与的电信级嵌入式实时操作系统,获得了行业最高奖----中国工业大奖。同时,他也是中国开源软件推进联盟专家委员会委员,Linux ZTE架构的Maintainer,向Linux提交了4000多行代码。他翻译的《深入理解并行编程》一书已经于2017年出版。该书原作者PaulE.McKeney是IBM Linux中心领导者、Linux RCU Maintainer。目前,他在维护阿里集团数以万计生产服务器稳定性的同时,也在利用业余时间继续编写DIM-SUM操作系统,并根据此操作系统编写《国产自研操作系统设计与实现》一书。
0 1 Load指标的意义Linux的Load指标代表系统运行负载,是一个让新手不太容易了解的概念。简单的说,Load指标表示一段时间内,系统有多少个正在运行的任务。包含:
1、正在CPU上面运行的任务数量
2、等待CPU调度运行的任务数量
3、等待IO执行结果的任务数量
根据工程经验,一般来说,当系统的Load指标值大于系统中CPU 数量的1.5~2倍时,表明系统负载较重,需要分析原因,采取措施降低Load指标值。否则可能产生系统卡顿的现象。如果系统的Load指标值大于系统中CPU 数量的3倍时,,表明系统负载非常严重。
随着多核系统中CPU数量的增加,以及容器虚拟化技术的发展,不同业务均运行在同一系统中,不可避免的造成业务之间的相互影响。此时,业务方常常会将系统Load指标作为衡量系统负载的重要指标。如果系统负载超过警告值,就会采取业务迁移等运维手段。并要求技术工程师分析Load高的原因。
因此,技术工程师有必要掌握Load高的分析手段。
0 2 如何观察Load指标我们一般通过top、uptime命令查看最近1分钟、5分钟、15分钟的平均Load指标。
下图是top命令的输出示例:
top - 13:54:54 up 1:49, 1 user, load average: 2.18, 1.72, 1.00
Tasks: 201 total, 4 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu(s): 8.0 us, 1.7 sy, 0.1 ni, 90.1 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 4039160 total, 624264 free, 1616064 used, 1798832 buff/cache
KiB Swap: 1045500 total, 0031045500 free, 0 used. 2007432 avail Mem
红色部分表示系统在最近1分钟、5分钟、15分钟的平均Load指标为2.18/1.72/1.00。
Load指标受到CPU利用率和IO等待时间的影响。
首先看看CPU利用率。在上图中,通过观察如下三个指标来判断CPU利用率:
%Cpu(s): 8.0 us, 1.7 sy, 0.1 ni, 90.1 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
这三个指标分别代表程序运行在用户态、内核态、以及运行在nice值调整期间的时间。您可以简单的认为这三个值之和代表了系统的CPU利用率。
如果系统中I/O密集型的程序在运行,那么iowait指标会偏高:
%Cpu(s): 8.0 us, 1.7 sy, 0.1 ni, 90.1 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
0 3如何构造测试用例首先,我们在控制台中运行如下命令,共计运行4次:
while :;do :;done &
这会启动4个后台运行的CPU密集型任务,占用4个CPU。
然后,使用如下命令启动fio,运行IO密集型任务:
sudo sh -c "while :; do fio -name=mytest -filename=test.data -direct=0 -thread -rw=randread -ioengine=psync -bs=4k -size=1M -numjobs=2 -runtime=10 -group_reporting;echo 1 > /proc/sys/vm/drop_caches; done"
我们可以在top命令中看到如下结果:
top - 14:31:34 up 2:26, 1 user, load average: 6.51, 5.14, 3.93Tasks: 206 total, 7 running, 156 sleeping, 0 stopped, 0 zombie%Cpu(s): 86.3 us, 12.9 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 stKiB Mem : 4039160 total, 1382444 free, 1599384 used, 1057332 buff/cacheKiB Swap: 1045500 total, 1045500 free, 0 used. 1743056 avail Mem
思考题:我们明明启动了一个fio这样的IO密集型任务,为什么上图中的wa指标为0?
有不少排查Load问题的方法,然而本文介绍的方法可以用于工程实践中,原因是:
1、轻量级,对系统性能的影响微乎其微
2、不需要安装第三方工具,易于部署
3、可以大规模部署,长时间运行
4、灵活,可以根据实际情况随时调整工具
接下来,我们看看如何一步一步的找到Load高问题的元凶!
0 4一步一步编写工具我们的工具是一个内核模块,首先实现一个最简单的模块,包含三个文件:
1、Makefile文件
OS_VER := UNKNOWNUNAME := $(shell uname -r)ifneq ($(findstring 4.15.0-39-generic,$(UNAME)),) OS_VER := UBUNTU_1604endififneq ($(KERNELRELEASE),) obj-m += $(MODNAME).o $(MODNAME)-y := main.o ccflags-y := -I$(PWD)/else export PWD=`pwd`ifeq ($(KERNEL_BUILD_PATH),) KERNEL_BUILD_PATH := /lib/modules/`uname -r`/buildendififeq ($(MODNAME),) export MODNAME=load_monitorendifall: make CFLAGS_MODULE=-D$(OS_VER) -C /lib/modules/`uname -r`/build M=`pwd` modulesclean: make -C $(KERNEL_BUILD_PATH) M=$(PWD) cleanendif
思考题:为什么要定义OS_VER这个变量?
2、main.c文件
/** * Baoyou Xie's load monitor module * * Copyright (C) 2018 Baoyou Xie. * * Author: Baoyou Xie * * License terms: GNU General Public License (GPL) version 2 */#include #include #include "load.h"static int load_monitor_init(void){ printk("load-monitor loaded.\n"); return 0;}static void load_monitor_exit(void){ printk("load-monitor unloaded.\n");}module_init(load_monitor_init)module_exit(load_monitor_exit)MODULE_DESCRIPTION("Baoyou Xie's load monitor module");MODULE_AUTHOR("Baoyou Xie ");MODULE_LICENSE("GPL v2");
请大家注意文件中的注释,以及文件末尾的模块说明。
3、load.h文件
/** * Baoyou Xie's load monitor module * * Copyright (C) 2018 Baoyou Xie. * * Author: Baoyou Xie @gmail.com> * * License terms: GNU General Public License (GPL) version 2 */
思考题:load.h里面什么都没有,这有什么用?
在源码目录下运行make命令,编译出load_monitor.ko模块文件。
运行如下命令,插入/卸载模块:
insmod load_monitor.ko; rmmod load_monitor.ko
然后使用如下命令查看模块输出:
root# dmesg -c[ 9984.791895] load-monitor loaded.[ 9984.794608] load-monitor unloaded.
目前,我们的模块还没有任何实际意义。
现在,我们一步一步的添加功能。
首先,我们在系统中添加一个定时器,该定时器每10毫秒执行一次。在main.c中,主要进行如下修改:
struct hrtimer timer;static void check_load(void){ printk("debug in check_load\n");}static enum hrtimer_restart monitor_handler(struct hrtimer *hrtimer){ enum hrtimer_restart ret = HRTIMER_RESTART; check_load(); hrtimer_forward_now(hrtimer, ms_to_ktime(10)); return ret;}static void start_timer(void){ hrtimer_init(&timer, CLOCK_MONOTONIC, HRTIMER_MODE_PINNED); timer.function = monitor_handler; hrtimer_start_range_ns(&timer, ms_to_ktime(10), 0, HRTIMER_MODE_REL_PINNED);}static int load_monitor_init(void){ start_timer(); printk("load-monitor loaded.\n"); return 0;}static void load_monitor_exit(void){ hrtimer_cancel(&timer); printk("load-monitor unloaded.\n");}
这仅仅是启动了一个定时器,并在定时器中简单的打印一句话。
思考题:这么久了,还没有切入正题。作者您是不是忘记了是在解Load高的问题?
运行如下命令,插入/卸载模块:
insmod load_monitor.ko;sleep 0.5; rmmod load_monitor.ko
然后运行如下命令,查看效果:
root# dmesg -c[ 169.840276] load-monitor loaded.[ 169.850579] debug in check_load..........[ 170.343457] load-monitor unloaded.
最关键的一步,我们需要读取系统的Load指标。Linux内核使用如下变量保存最近1分钟、5分钟、15分钟的Load值:
unsigned long avenrun[3];
虽然我们可以在模块中直接读取这个值。但是为了防止某些版本的内核没有通过EXPORT_SYMBOL导出这个变量,我们使用如下语句获得这个变量的地址:
unsigned long *ptr_avenrun;ptr_avenrun = (void *)kallsyms_lookup_name("avenrun");
思考题:某些版本的Linux内核不能直接调用kallsyms_lookup_name,有哪些办法获得内核函数/变量的地址?
修改后的代码如下:
static unsigned long *ptr_avenrun;#define FSHIFT 11 /* nr of bits of precision */#define FIXED_1 (1</* 1.0 as fixed-point */#define LOAD_INT(x) ((x) >> FSHIFT)#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)static void print_all_task_stack(void){ printk("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n"); printk("Load: %lu.%02lu, %lu.%02lu, %lu.%02lu\n", LOAD_INT(ptr_avenrun[0]), LOAD_FRAC(ptr_avenrun[0]), LOAD_INT(ptr_avenrun[1]), LOAD_FRAC(ptr_avenrun[1]), LOAD_INT(ptr_avenrun[2]), LOAD_FRAC(ptr_avenrun[2])); printk("dump all task: balabala\n");}static void check_load(void){ static ktime_t last; u64 ms; int load = LOAD_INT(ptr_avenrun[0]); /* 最近1分钟的Load值 */ if (load < 4) return; /** * 如果上次打印时间与当前时间相差不到20秒,就直接退出 */ ms = ktime_to_ms(ktime_sub(ktime_get(), last)); if (ms < 20 * 1000) return; last = ktime_get(); print_all_task_stack();}static enum hrtimer_restart monitor_handler(struct hrtimer *hrtimer){ enum hrtimer_restart ret = HRTIMER_RESTART; check_load(); hrtimer_forward_now(hrtimer, ms_to_ktime(10)); return ret;}
重新加载模块,并使用dmesg命令,可以得到如下结果:
[ 2300.730884] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![ 2300.730888] Load: 4.13, 1.51, 0.66[ 2300.730888] dump all task: balabala
我们已经能够探测到Load升高的情况了。现在我们只需要完善一下print_all_task_stack函数,输出所有线程的堆栈即可。
修改后的print_all_task_stack函数如下:
static void print_all_task_stack(void){ struct task_struct *g, *p; unsigned long backtrace[BACKTRACE_DEPTH]; struct stack_trace trace; memset(&trace, 0, sizeof(trace)); memset(backtrace, 0, BACKTRACE_DEPTH * sizeof(unsigned long)); trace.max_entries = BACKTRACE_DEPTH; trace.entries = backtrace; printk("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n"); printk("\tLoad: %lu.%02lu, %lu.%02lu, %lu.%02lu\n", LOAD_INT(ptr_avenrun[0]), LOAD_FRAC(ptr_avenrun[0]), LOAD_INT(ptr_avenrun[1]), LOAD_FRAC(ptr_avenrun[1]), LOAD_INT(ptr_avenrun[2]), LOAD_FRAC(ptr_avenrun[2])); rcu_read_lock(); printk("dump running task.\n"); do_each_thread(g, p) { if (p->state == TASK_RUNNING) { printk("running task, comm: %s, pid %d\n", p->comm, p->pid); save_stack_trace_tsk(p, &trace); print_stack_trace(&trace, 0); } } while_each_thread(g, p); printk("dump uninterrupted task.\n"); do_each_thread(g, p) { if (p->state & TASK_UNINTERRUPTIBLE) { printk("uninterrupted task, comm: %s, pid %d\n", p->comm, p->pid); save_stack_trace_tsk(p, &trace); print_stack_trace(&trace, 0); } } while_each_thread(g, p); rcu_read_unlock();}
重新编译,插入/卸载模块,并使用dmesg -c查看输出。可以看到如下调用链。
您可以在这些堆栈中找到名为bash和fio的线程。这些线程正是我们测试用例启动的线程。
[ 6140.570181] !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![ 6140.570185] Load: 5.48, 5.80, 5.60[ 6140.570185] dump running task.[ 6140.570214] running task, comm: llvmpipe-0, pid 2437[ 6140.570225] futex_wait_queue_me+0xd6/0x120[ 6140.570227] futex_wait+0x240/0x260[ 6140.570229] do_futex+0x128/0x500[ 6140.570231] SyS_futex+0x7f/0x180[ 6140.570234] do_syscall_64+0x73/0x130[ 6140.570239] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570240] 0xffffffffffffffff[ 6140.570242] running task, comm: llvmpipe-1, pid 2438[ 6140.570244] exit_to_usermode_loop+0x66/0xd0[ 6140.570245] prepare_exit_to_usermode+0x5a/0x80[ 6140.570247] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570248] 0xffffffffffffffff[ 6140.570249] running task, comm: llvmpipe-2, pid 2439[ 6140.570251] exit_to_usermode_loop+0x66/0xd0[ 6140.570252] prepare_exit_to_usermode+0x5a/0x80[ 6140.570254] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570255] 0xffffffffffffffff[ 6140.570255] running task, comm: llvmpipe-3, pid 2440[ 6140.570257] exit_to_usermode_loop+0x66/0xd0[ 6140.570258] prepare_exit_to_usermode+0x5a/0x80[ 6140.570260] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570261] 0xffffffffffffffff[ 6140.570269] running task, comm: bash, pid 4687[ 6140.570271] exit_to_usermode_loop+0x66/0xd0[ 6140.570273] prepare_exit_to_usermode+0x5a/0x80[ 6140.570274] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570275] 0xffffffffffffffff[ 6140.570276] running task, comm: bash, pid 4688[ 6140.570287] exit_to_usermode_loop+0x66/0xd0[ 6140.570288] prepare_exit_to_usermode+0x5a/0x80[ 6140.570290] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570291] 0xffffffffffffffff[ 6140.570291] running task, comm: bash, pid 4689[ 6140.570293] exit_to_usermode_loop+0x66/0xd0[ 6140.570295] prepare_exit_to_usermode+0x5a/0x80[ 6140.570296] swapgs_restore_regs_and_return_to_usermode+0x0/0x87[ 6140.570297] 0xffffffffffffffff[ 6140.570298] running task, comm: bash, pid 4690[ 6140.570303] print_all_task_stack+0x17f/0x270 [load_monitor][ 6140.570305] monitor_handler+0x86/0x90 [load_monitor][ 6140.570307] __hrtimer_run_queues+0xe7/0x230[ 6140.570308] hrtimer_interrupt+0xb1/0x200[ 6140.570309] smp_apic_timer_interrupt+0x6f/0x140[ 6140.570311] apic_timer_interrupt+0x84/0x90[ 6140.570311] 0xffffffffffffffff[ 6140.570313] running task, comm: sh, pid 4703[ 6140.570317] invalidate_mapping_pages+0x23e/0x330[ 6140.570320] drop_pagecache_sb+0xb2/0xf0[ 6140.570323] iterate_supers+0xbb/0x110[ 6140.570324] drop_caches_sysctl_handler+0x51/0xb0[ 6140.570327] proc_sys_call_handler+0xf1/0x110[ 6140.570328] proc_sys_write+0x14/0x20[ 6140.570329] __vfs_write+0x1b/0x40[ 6140.570330] vfs_write+0xb8/0x1b0[ 6140.570331] SyS_write+0x55/0xc0[ 6140.570333] do_syscall_64+0x73/0x130[ 6140.570334] entry_SYSCALL_64_after_hwframe+0x3d/0xa2[ 6140.570335] 0xffffffffffffffff[ 6140.570338] running task, comm: insmod, pid 10903[ 6140.570340] __blocking_notifier_call_chain+0x32/0x60[ 6140.570341] blocking_notifier_call_chain+0x16/0x20[ 6140.570343] do_init_module+0xa4/0x219[ 6140.570345] load_module+0x1937/0x1d80[ 6140.570346] SYSC_finit_module+0xe5/0x120[ 6140.570347] SyS_finit_module+0xe/0x10[ 6140.570349] do_syscall_64+0x73/0x130[ 6140.570350] entry_SYSCALL_64_after_hwframe+0x3d/0xa2[ 6140.570351] 0xffffffffffffffff[ 6140.570352] dump uninterrupted task.[ 6140.570353] uninterrupted task, comm: kworker/0:0, pid 3[ 6140.570355] worker_thread+0xc9/0x460[ 6140.570357] kthread+0x105/0x140[ 6140.570358] ret_from_fork+0x35/0x40[ 6140.570359] 0xffffffffffffffff[ 6140.570360] uninterrupted task, comm: kworker/0:0H, pid 4[ 6140.570361] worker_thread+0xc9/0x460[ 6140.570363] kthread+0x105/0x140[ 6140.570364] ret_from_fork+0x35/0x40[ 6140.570365] 0xffffffffffffffff[ 6140.570366] uninterrupted task, comm: mm_percpu_wq, pid 6[ 6140.570368] rescuer_thread+0x337/0x380[ 6140.570369] kthread+0x105/0x140[ 6140.570370] ret_from_fork+0x35/0x40[ 6140.570371] 0xffffffffffffffff[ 6140.570372] uninterrupted task, comm: rcu_sched, pid 8[ 6140.570375] rcu_gp_kthread+0x5b4/0x960[ 6140.570376] kthread+0x105/0x140[ 6140.570378] ret_from_fork+0x35/0x40[ 6140.570378] 0xffffffffffffffff[ 6140.570379] uninterrupted task, comm: rcu_bh, pid 9[ 6140.570380] rcu_gp_kthread+0x83/0x960[ 6140.570382] kthread+0x105/0x140[ 6140.570383] ret_from_fork+0x35/0x40[ 6140.570384] 0xffffffffffffffff[ 6140.570385] uninterrupted task, comm: kworker/1:0H, pid 18[ 6140.570387] worker_thread+0xc9/0x460[ 6140.570388] kthread+0x105/0x140[ 6140.570390] ret_from_fork+0x35/0x40[ 6140.570390] 0xffffffffffffffff[ 6140.570391] uninterrupted task, comm: kworker/2:0H, pid 24[ 6140.570393] worker_thread+0xc9/0x460[ 6140.570394] kthread+0x105/0x140[ 6140.570396] ret_from_fork+0x35/0x40[ 6140.570396] 0xffffffffffffffff[ 6140.570397] uninterrupted task, comm: kworker/3:0H, pid 30[ 6140.570399] worker_thread+0xc9/0x460[ 6140.570400] kthread+0x105/0x140[ 6140.570402] ret_from_fork+0x35/0x40[ 6140.570403] 0xffffffffffffffff[ 6140.570403] uninterrupted task, comm: netns, pid 32[ 6140.570405] rescuer_thread+0x337/0x380[ 6140.570407] kthread+0x105/0x140[ 6140.570408] ret_from_fork+0x35/0x40[ 6140.570409] 0xffffffffffffffff[ 6140.570410] uninterrupted task, comm: writeback, pid 38[ 6140.570412] rescuer_thread+0x337/0x380[ 6140.570413] kthread+0x105/0x140[ 6140.570415] ret_from_fork+0x35/0x40[ 6140.570415] 0xffffffffffffffff[ 6140.570416] uninterrupted task, comm: crypto, pid 42[ 6140.570418] rescuer_thread+0x337/0x380[ 6140.570420] kthread+0x105/0x140[ 6140.570421] ret_from_fork+0x35/0x40[ 6140.570422] 0xffffffffffffffff[ 6140.570423] uninterrupted task, comm: kintegrityd, pid 43[ 6140.570424] rescuer_thread+0x337/0x380[ 6140.570426] kthread+0x105/0x140[ 6140.570427] ret_from_fork+0x35/0x40[ 6140.570428] 0xffffffffffffffff[ 6140.570429] uninterrupted task, comm: kblockd, pid 44[ 6140.570431] rescuer_thread+0x337/0x380[ 6140.570432] kthread+0x105/0x140[ 6140.570434] ret_from_fork+0x35/0x40[ 6140.570434] 0xffffffffffffffff[ 6140.570435] uninterrupted task, comm: ata_sff, pid 45[ 6140.570437] rescuer_thread+0x337/0x380[ 6140.570439] kthread+0x105/0x140[ 6140.570440] ret_from_fork+0x35/0x40[ 6140.570441] 0xffffffffffffffff[ 6140.570441] uninterrupted task, comm: md, pid 46[ 6140.570443] rescuer_thread+0x337/0x380[ 6140.570445] kthread+0x105/0x140[ 6140.570446] ret_from_fork+0x35/0x40[ 6140.570447] 0xffffffffffffffff[ 6140.570447] uninterrupted task, comm: edac-poller, pid 47[ 6140.570449] rescuer_thread+0x337/0x380[ 6140.570468] kthread+0x105/0x140[ 6140.570470] ret_from_fork+0x35/0x40[ 6140.570471] 0xffffffffffffffff[ 6140.570472] uninterrupted task, comm: devfreq_wq, pid 48[ 6140.570474] rescuer_thread+0x337/0x380[ 6140.570475] kthread+0x105/0x140[ 6140.570477] ret_from_fork+0x35/0x40[ 6140.570477] 0xffffffffffffffff[ 6140.570478] uninterrupted task, comm: watchdogd, pid 49[ 6140.570480] rescuer_thread+0x337/0x380[ 6140.570481] kthread+0x105/0x140[ 6140.570483] ret_from_fork+0x35/0x40[ 6140.570483] 0xffffffffffffffff[ 6140.570484] uninterrupted task, comm: kworker/3:1, pid 58[ 6140.570486] worker_thread+0xc9/0x460[ 6140.570488] kthread+0x105/0x140[ 6140.570489] ret_from_fork+0x35/0x40[ 6140.570490] 0xffffffffffffffff[ 6140.570490] uninterrupted task, comm: kworker/1:1, pid 67[ 6140.570492] worker_thread+0xc9/0x460[ 6140.570494] kthread+0x105/0x140[ 6140.570495] ret_from_fork+0x35/0x40[ 6140.570496] 0xffffffffffffffff[ 6140.570496] uninterrupted task, comm: kthrotld, pid 98[ 6140.570498] rescuer_thread+0x337/0x380[ 6140.570500] kthread+0x105/0x140[ 6140.570501] ret_from_fork+0x35/0x40[ 6140.570502] 0xffffffffffffffff[ 6140.570503] uninterrupted task, comm: acpi_thermal_pm, pid 99[ 6140.570505] rescuer_thread+0x337/0x380[ 6140.570506] kthread+0x105/0x140[ 6140.570507] ret_from_fork+0x35/0x40[ 6140.570508] 0xffffffffffffffff[ 6140.570509] uninterrupted task, comm: scsi_tmf_0, pid 101[ 6140.570555] rescuer_thread+0x337/0x380[ 6140.570558] kthread+0x105/0x140[ 6140.570561] ret_from_fork+0x35/0x40[ 6140.570563] 0xffffffffffffffff[ 6140.570565] uninterrupted task, comm: scsi_tmf_1, pid 103[ 6140.570568] rescuer_thread+0x337/0x380[ 6140.570570] kthread+0x105/0x140[ 6140.570572] ret_from_fork+0x35/0x40[ 6140.570573] 0xffffffffffffffff[ 6140.570575] uninterrupted task, comm: ipv6_addrconf, pid 110[ 6140.570577] rescuer_thread+0x337/0x380[ 6140.570579] kthread+0x105/0x140[ 6140.570582] ret_from_fork+0x35/0x40[ 6140.570583] 0xffffffffffffffff[ 6140.570584] uninterrupted task, comm: kstrp, pid 119[ 6140.570586] rescuer_thread+0x337/0x380[ 6140.570588] kthread+0x105/0x140[ 6140.570590] ret_from_fork+0x35/0x40[ 6140.570591] 0xffffffffffffffff[ 6140.570592] uninterrupted task, comm: charger_manager, pid 136[ 6140.570595] rescuer_thread+0x337/0x380[ 6140.570597] kthread+0x105/0x140[ 6140.570599] ret_from_fork+0x35/0x40[ 6140.570600] 0xffffffffffffffff[ 6140.570601] uninterrupted task, comm: kworker/2:2, pid 176[ 6140.570604] worker_thread+0xc9/0x460[ 6140.570606] kthread+0x105/0x140[ 6140.570608] ret_from_fork+0x35/0x40[ 6140.570609] 0xffffffffffffffff[ 6140.570610] uninterrupted task, comm: kworker/1:2, pid 195[ 6140.570613] worker_thread+0xc9/0x460[ 6140.570615] kthread+0x105/0x140[ 6140.570617] ret_from_fork+0x35/0x40[ 6140.570618] 0xffffffffffffffff[ 6140.570619] uninterrupted task, comm: scsi_tmf_2, pid 198[ 6140.570622] rescuer_thread+0x337/0x380[ 6140.570624] kthread+0x105/0x140[ 6140.570626] ret_from_fork+0x35/0x40[ 6140.570627] 0xffffffffffffffff[ 6140.570628] uninterrupted task, comm: kworker/2:1H, pid 199[ 6140.570631] worker_thread+0xc9/0x460[ 6140.570633] kthread+0x105/0x140[ 6140.570635] ret_from_fork+0x35/0x40[ 6140.570636] 0xffffffffffffffff[ 6140.570637] uninterrupted task, comm: kworker/0:1H, pid 201[ 6140.570640] worker_thread+0xc9/0x460[ 6140.570642] kthread+0x105/0x140[ 6140.570644] ret_from_fork+0x35/0x40[ 6140.570645] 0xffffffffffffffff[ 6140.570646] uninterrupted task, comm: kworker/3:1H, pid 211[ 6140.570648] worker_thread+0xc9/0x460[ 6140.570650] kthread+0x105/0x140[ 6140.570652] ret_from_fork+0x35/0x40[ 6140.570653] 0xffffffffffffffff[ 6140.570654] uninterrupted task, comm: kworker/1:1H, pid 215[ 6140.570657] worker_thread+0xc9/0x460[ 6140.570659] kthread+0x105/0x140[ 6140.570661] ret_from_fork+0x35/0x40[ 6140.570662] 0xffffffffffffffff[ 6140.570663] uninterrupted task, comm: ext4-rsv-conver, pid 227[ 6140.570666] rescuer_thread+0x337/0x380[ 6140.570668] kthread+0x105/0x140[ 6140.570669] ret_from_fork+0x35/0x40[ 6140.570671] 0xffffffffffffffff[ 6140.570672] uninterrupted task, comm: iprt-VBoxWQueue, pid 328[ 6140.570674] rescuer_thread+0x337/0x380[ 6140.570676] kthread+0x105/0x140[ 6140.570678] ret_from_fork+0x35/0x40[ 6140.570679] 0xffffffffffffffff[ 6140.570681] uninterrupted task, comm: ttm_swap, pid 407[ 6140.570683] rescuer_thread+0x337/0x380[ 6140.570685] kthread+0x105/0x140[ 6140.570687] ret_from_fork+0x35/0x40[ 6140.570688] 0xffffffffffffffff[ 6140.570702] uninterrupted task, comm: kworker/0:1, pid 3230[ 6140.570705] worker_thread+0xc9/0x460[ 6140.570707] kthread+0x105/0x140[ 6140.570709] ret_from_fork+0x35/0x40[ 6140.570710] 0xffffffffffffffff[ 6140.570712] uninterrupted task, comm: kworker/u8:2, pid 3804[ 6140.570714] worker_thread+0xc9/0x460[ 6140.570716] kthread+0x105/0x140[ 6140.570718] ret_from_fork+0x35/0x40[ 6140.570719] 0xffffffffffffffff[ 6140.570720] uninterrupted task, comm: kworker/3:0, pid 3838[ 6140.570723] worker_thread+0xc9/0x460[ 6140.570725] kthread+0x105/0x140[ 6140.570727] ret_from_fork+0x35/0x40[ 6140.570728] 0xffffffffffffffff[ 6140.570730] uninterrupted task, comm: kworker/u8:0, pid 4702[ 6140.570732] worker_thread+0xc9/0x460[ 6140.570734] kthread+0x105/0x140[ 6140.570736] ret_from_fork+0x35/0x40[ 6140.570738] 0xffffffffffffffff[ 6140.570739] uninterrupted task, comm: kworker/2:0, pid 17180[ 6140.570741] worker_thread+0xc9/0x460[ 6140.570743] kthread+0x105/0x140[ 6140.570745] ret_from_fork+0x35/0x40[ 6140.570746] 0xffffffffffffffff[ 6140.570747] uninterrupted task, comm: kworker/u8:3, pid 23671[ 6140.570750] worker_thread+0xc9/0x460[ 6140.570752] kthread+0x105/0x140[ 6140.570754] ret_from_fork+0x35/0x40[ 6140.570755] 0xffffffffffffffff[ 6140.600832] sh (4703): drop_caches: 1
思考题:模块代码还有哪些值得改进的地方?
思考题:您还有其他方法跟踪Load高问题吗?不同的方法什么优缺点?
说明:该内容是谢宝友老师为学堂在线的《Linux内核分析与应用》慕课而撰写,对应的视频是进程管理中的3.5节-工程实践-基于内核模块的负载监控,大家可以移步学堂在线学习《Linux内核分析与应用》
目前100000+人已关注加入我们