[exception_reboot_reason]: kernel_crash
[exception_panic_reason]: Native hang monitor trigger
结论:
需要内存管理的同事在帮忙check下
代码中确实存在down_read后，handle_mm_fault - handle_pte_fault - do_wp_page - balance_dirty_pages_ratelimited - schedule
当前UN进程里存在
ffffffc00e7773f0
ffffffc0144652c8
ffffffc004cf9ff0
ffffffc004cf91f0
ffffffc004cfdb70
ffffffc00e7773f0 ffffffc004cfdb70 和 ffffffc004cf91f0 ffffffc004cf91f0d都被balance_dirty_pages_ratelimited出去。
ffffffc0144652c8被15577 block balance_dirty_pages_ratelimited - schedule
推导过程:
crash_arm64> rd cpu_feed_mask
ffffff8009a9d0e0: 00000000000000ff ........
crash_arm64> rd cpu_feed_bitmap
ffffff8009a9d0e4: d2fbd2fb00000000 ........
crash_arm64> ps -l | grep hang
[119352 286 360 532] [RU] PID: 607 TASK: ffffffc0a5120e80 CPU: 0 COMMAND: "native_hang_det"
[119348327880470] [IN] PID: 209 TASK: ffffffc0b9fec880 CPU: 7 COMMAND: "hang_debug/7"
[119348325009162] [IN] PID: 208 TASK: ffffffc0b9fed700 CPU: 6 COMMAND: "hang_debug/6"
[119348322574624] [IN] PID: 207 TASK: ffffffc0b9fee580 CPU: 5 COMMAND: "hang_debug/5"
[119348320191931] [IN] PID: 206 TASK: ffffffc0b9ff0000 CPU: 4 COMMAND: "hang_debug/4"
[119348317873508] [IN] PID: 205 TASK: ffffffc0b9ff0e80 CPU: 3 COMMAND: "hang_debug/3"
[119348315890085] [IN] PID: 204 TASK: ffffffc0b9ff1d00 CPU: 2 COMMAND: "hang_debug/2"
[119348313697431] [IN] PID: 203 TASK: ffffffc0b9ff2b80 CPU: 1 COMMAND: "hang_debug/1"
[119345025989944] [IN] PID: 202 TASK: ffffffc0b9ff3a00 CPU: 0 COMMAND: "hang_debug/0"
疑点2:
[119352298356416] [UN] PID: 21715 TASK: ffffffc036a90e80 CPU: 6 COMMAND: "AsyncTask #95"
[119352298349570] [UN] PID: 1076 TASK: ffffffc07b911d00 CPU: 1 COMMAND: "android.bg"
[119352278352686] [UN] PID: 2648 TASK: ffffffc0951b0000 CPU: 0 COMMAND: "DFacilitator-1"
[119352274368378] [UN] PID: 1054 TASK: ffffffc095020e80 CPU: 7 COMMAND: "Signal Catcher"
[119352266347032] [UN] PID: 14286 TASK: ffffffc0acebba00 CPU: 0 COMMAND: "slogmodem"
[119352258363647] [UN] PID: 15577 TASK: ffffffc08573d700 CPU: 7 COMMAND: "Thread-18"
[119352258342455] [UN] PID: 4749 TASK: ffffffc083183a00 CPU: 6 COMMAND: "lowpool[366]"
[119352254350070] [UN] PID: 2779 TASK: ffffffc008381d00 CPU: 0 COMMAND: "ogle.android.gm"
[119352238358917] [UN] PID: 3213 TASK: ffffffc04e279d00 CPU: 0 COMMAND: "HangWatcher"
[119352234402378] [UN] PID: 21746 TASK: ffffffc013fcba00 CPU: 6 COMMAND: "queued-work-loo"
[119352234367609] [UN] PID: 10040 TASK: ffffffc08630d700 CPU: 6 COMMAND: "Alpha Thread #2"
[119352202370955] [UN] PID: 21924 TASK: ffffffc084331d00 CPU: 7 COMMAND: "CrashSDKBkgdHan"
[119352202353570] [UN] PID: 14290 TASK: ffffffc00381e580 CPU: 0 COMMAND: "slogmodem"
[119352182371609] [UN] PID: 15622 TASK: ffffffc0953ae580 CPU: 7 COMMAND: "ageEmmcTestThre"
[119352182367724] [UN] PID: 3216 TASK: ffffffc01ae01d00 CPU: 0 COMMAND: "ThreadPoolForeg"
[119352158388378] [UN] PID: 4830 TASK: ffffffc00fa8c880 CPU: 0 COMMAND: "Chrome_IOThread"
[119352138372417] [UN] PID: 3150 TASK: ffffffc01b0a8000 CPU: 0 COMMAND: ".android.chrome"
[119352134414494] [UN] PID: 2207 TASK: ffffffc015b6c880 CPU: 7 COMMAND: "queued-work-loo"
[119352134383686] [UN] PID: 17671 TASK: ffffffc01618d700 CPU: 7 COMMAND: "Alpha Thread #6"
[119352134374186] [UN] PID: 14282 TASK: ffffffc03b890000 CPU: 2 COMMAND: "slogmodem"
[119352122370648] [UN] PID: 1048 TASK: ffffffc099e50000 CPU: 7 COMMAND: "system_server"
[119352122361186] [UN] PID: 707 TASK: ffffffc0a0d20e80 CPU: 2 COMMAND: "slogmodem"
[119352118368225] [UN] PID: 471 TASK: ffffffc0b2090e80 CPU: 2 COMMAND: "ylog.journal"
[119352118346302] [UN] PID: 14923 TASK: ffffffc01ae04880 CPU: 0 COMMAND: "acji"
[119352110370225] [UN] PID: 23223 TASK: ffffffc00272ab80 CPU: 0 COMMAND: "dumpstate"
[119352098459648] [UN] PID: 21876 TASK: ffffffc00452e580 CPU: 7 COMMAND: "Alpha Simple Th"
[119352098376763] [UN] PID: 1068 TASK: ffffffc07bb91d00 CPU: 7 COMMAND: "android.io"
[119352094398571] [UN] PID: 631 TASK: ffffffc0a52fab80 CPU: 0 COMMAND: "ylog.writer"
crash_arm64> bt 631
PID: 631 TASK: ffffffc0a52fab80 CPU: 0 COMMAND: "ylog.writer"
#0 [ffffff800e0bb7d0] __switch_to at ffffff80081f4ae0
#1 [ffffff800e0bb820] __schedule at ffffff8008e1518c
#2 [ffffff800e0bb880] schedule at ffffff8008e15528
#3 [ffffff800e0bb900] schedule_timeout at ffffff8008e193cc
#4 [ffffff800e0bb960] io_schedule_timeout at ffffff8008e15a74
#5 [ffffff800e0bbb50] balance_dirty_pages_ratelimited at ffffff800838bd24
#6 [ffffff800e0bbbc0] do_wp_page at ffffff80083c3ba0
#7 [ffffff800e0bbc10] handle_pte_fault at ffffff80083c4c90
#8 [ffffff800e0bbcf0] handle_mm_fault at ffffff80083c5b70
#9 [ffffff800e0bbd90] do_page_fault$2ad2e1b83c15cf95a0f028b354d101f0 at ffffff8008210510
#10 [ffffff800e0bbe80] do_mem_abort at ffffff80080815dc
#11 [ffffff800e0bbff0] el0_da at ffffff8008084054
[119346636502361] [UN] PID: 21811 TASK: ffffffc015904880 CPU: 6 COMMAND: "Thread-5442" -- rwsem_down_write_failed_killable - ffffffc004cfdb70
[119326638527706] [UN] PID: 21812 TASK: ffffffc013540000 CPU: 6 COMMAND: "Thread-5446" -- rwsem_down_write_failed_killable - ffffffc004cfdb70
[119284381064598] [UN] PID: 5567 TASK: ffffffc084db0000 CPU: 7 COMMAND: "grpc-timer-0" -- rwsem_down_read_failed - ffffffc00e7773f0
[119225439135477] [UN] PID: 10033 TASK: ffffffc012668e80 CPU: 6 COMMAND: "Thread-2763" -- rwsem_down_write_failed_killable - ffffffc004cf91f0
[119225439128746] [UN] PID: 10032 TASK: ffffffc012668000 CPU: 7 COMMAND: "Thread-2761" -- rwsem_down_write_failed_killable - ffffffc004cf91f0
[119205590461168] [UN] PID: 4828 TASK: ffffffc004c80000 CPU: 2 COMMAND: "ThreadPoolForeg" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[119120751987723] [UN] PID: 17642 TASK: ffffffc083186580 CPU: 7 COMMAND: "AppMonitor:0" -- rwsem_down_read_failed - ffffffc004cf9ff0
[119082242871562] [UN] PID: 3035 TASK: ffffffc083160e80 CPU: 0 COMMAND: "ThreadPoolForeg" -- rwsem_down_write_failed_killable - ffffffc004cfe5f0
[119005150455356] [UN] PID: 3221 TASK: ffffffc088978e80 CPU: 2 COMMAND: "ThreadPoolForeg" -- rwsem_down_write_failed_killable - ffffffc004cfe270
[119001102348602] [UN] PID: 22849 TASK: ffffffc002140e80 CPU: 0 COMMAND: "ThreadPoolForeg" -- rwsem_down_write_failed_killable - ffffffc004cfe5f0
[118976530368311] [UN] PID: 19825 TASK: ffffffc017708000 CPU: 0 COMMAND: "UserFacingBlock" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118976526353619] [UN] PID: 19826 TASK: ffffffc017708e80 CPU: 1 COMMAND: "UserFacingBlock" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118960468503219] [UN] PID: 17657 TASK: ffffffc0137f6580 CPU: 7 COMMAND: "Thread-3304" -- rwsem_down_write_failed_killable - ffffffc004cf9ff0
[118958418145188] [UN] PID: 17597 TASK: ffffffc083161d00 CPU: 7 COMMAND: "Jit thread pool" -- rwsem_down_read_failed - ffffffc004cf9ff0
[118940471159295] [UN] PID: 17658 TASK: ffffffc0051f3a00 CPU: 7 COMMAND: "Thread-3294" -- rwsem_down_write_failed_killable - ffffffc004cf9ff0
[118862162523094] [UN] PID: 4833 TASK: ffffffc0083c1d00 CPU: 1 COMMAND: "ThreadPoolForeg" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
###############
[118836162877616] [UN] PID: 1072 TASK: ffffffc07bb95700 CPU: 6 COMMAND: "watchdog" -- rwsem_down_read_failed - ffffffc00e7773f0
#############
[118819464467680] [UN] PID: 15532 TASK: ffffffc0a533c880 CPU: 7 COMMAND: "ageTestThread" -- rwsem_down_write_failed - ffffffc0144652c8
[118812146819169] [UN] PID: 22143 TASK: ffffffc042224880 CPU: 0 COMMAND: "UserFacingBlock" -- rwsem_down_write_failed_killable -- ffffffc00e7773f0
[118782600367513] [UN] PID: 32074 TASK: ffffffc071fc8e80 CPU: 7 COMMAND: "HeapTaskDaemon" -- rwsem_down_read_failed - ffffffc00e7773f0
[118737434910031] [UN] PID: 22126 TASK: ffffffc08845ab80 CPU: 0 COMMAND: "Blocking Thread" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118717714142722] [UN] PID: 22402 TASK: ffffffc08041d700 CPU: 1 COMMAND: "Blocking Thread" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118714934390001] [UN] PID: 22621 TASK: ffffffc05f8e8e80 CPU: 2 COMMAND: "Blocking Thread" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118714934039078] [UN] PID: 22414 TASK: ffffffc002728000 CPU: 1 COMMAND: "Blocking Thread" --rwsem_down_write_failed_killable - ffffffc00e7773f0
[118679601158175] [UN] PID: 22697 TASK: ffffffc036bc0000 CPU: 7 COMMAND: "GAC_Executor[12" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118679600623482] [UN] PID: 22698 TASK: ffffffc002729d00 CPU: 6 COMMAND: "GAC_Executor[13" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118677477690491] [UN] PID: 32125 TASK: ffffffc0369f2b80 CPU: 0 COMMAND: "RenderThread" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118677440422375] [UN] PID: 32073 TASK: ffffffc01b730e80 CPU: 6 COMMAND: "Jit thread pool" -- rwsem_down_write_failed_killable - ffffffc00e7773f0
[118676 970 363 800] [UN] PID: 32130 TASK: ffffffc00473ab80 CPU: 0 COMMAND: "mali-mem-purge" -- down_write_killable ffffffc00e7773f0
watchdog进程
#3 [ffffff8012e6bcc0] rwsem_down_read_failed at ffffff8008e18644
ffffff8012e6bcc0: ffffff8012e6bd10 ffffff8008e17d7c x29, x30
ffffff8012e6bcd0: 000000000000003f 0000000000000fff x26, x25
ffffff8012e6bce0: 0000000000000015 000000774a7d47b8 x24, x23
ffffff8012e6bcf0: ffffffc00e7773f0 ffffffc00e777380 x22, x21
ffffff8012e6bd00: ffffffc001c38e80 ffffffc00e7773f0 x20, x19
crash_arm64> struct rw_semaphore ffffffc00e7773f0 -x
struct rw_semaphore {
count = {
counter = 0xffffffff00000001
},
wait_list = {
next = 0xffffff8012723dd8,
prev = 0xffffff800c9f3c88
},
wait_lock = {
raw_lock = {
owner = 0x1142,
next = 0x1142
}
},
osq = {
tail = {
counter = 0x0
}
},
owner = 0x1
}
crash_arm64> search -t ffffffc00e7773f0
search: invalid list entry: 0
search: WARNING: invalid/corrupt vmap_area_list
PID: 1072 TASK: ffffffc07bb95700 CPU: 6 COMMAND: "watchdog" //UN-ps
ffffff8012e6bc60: ffffffc00e7773f0
ffffff8012e6bc68: ffffffc00e7773f0
ffffff8012e6bcf0: ffffffc00e7773f0
ffffff8012e6bd08: ffffffc00e7773f0
PID: 4828 TASK: ffffffc004c80000 CPU: 2 COMMAND: "ThreadPoolForeg"//UN-ps
ffffff800d69bd20: ffffffc00e7773f0
ffffff800d69bd90: ffffffc00e7773f0
ffffff800d69bdc8: ffffffc00e7773f0
PID: 4830 TASK: ffffffc00fa8c880 CPU: 0 COMMAND: "Chrome_IOThread"//suspect
ffffff800a203b78: ffffffc00e7773f0
ffffff800a203c28: ffffffc00e7773f0
[119352158388378] [UN] PID: 4830 TASK: ffffffc00fa8c880 CPU: 0 COMMAND: "Chrome_IOThread"
PID: 4833 TASK: ffffffc0083c1d00 CPU: 1 COMMAND: "ThreadPoolForeg" //UN-ps
ffffff800d6bbda0: ffffffc00e7773f0
ffffff800d6bbe38: ffffffc00e7773f0
ffffff800d6bbe48: ffffffc00e7773f0
PID: 4838 TASK: ffffffc004c83a00 CPU: 2 COMMAND: "NetworkService" //fail
ffffff800d70b578: ffffffc00e7773f0
ffffff800d70b718: ffffffc00e7773f0
ffffff800d70b778: ffffffc00e7773f0
[118532 123 726 276] [IN] PID: 4838 TASK: ffffffc004c83a00 CPU: 2 COMMAND: "NetworkService"
PID: 4861 TASK: ffffffc006eb6580 CPU: 5 COMMAND: "Binder:32060_8" //binder
ffffff800dbe3478: ffffffc00e7773f0
[118611 910 015 241] [IN] PID: 4861 TASK: ffffffc006eb6580 CPU: 5 COMMAND: "Binder:32060_8"
PID: 5177 TASK: ffffffc00737ba00 CPU: 5 COMMAND: "Binder:32060_A" //binder
ffffff800c36b478: ffffffc00e7773f0
ffffff800c36b618: ffffffc00e7773f0
ffffff800c36b678: ffffffc00e7773f0
[118543653119770] [IN] PID: 5177 TASK: ffffffc00737ba00 CPU: 5 COMMAND: "Binder:32060_A"
PID: 5475 TASK: ffffffc0b9934880 CPU: 2 COMMAND: "Binder:32060_D" //binder
ffffff8010c2b478: ffffffc00e7773f0
ffffff8010c2b618: ffffffc00e7773f0
ffffff8010c2b678: ffffffc00e7773f0
[118543 652 690 193] [IN] PID: 5475 TASK: ffffffc0b9934880 CPU: 2 COMMAND: "Binder:32060_D"
PID: 5567 TASK: ffffffc084db0000 CPU: 7 COMMAND: "grpc-timer-0"//UN-ps
ffffff800c9f3c50: ffffffc00e7773f0
ffffff800c9f3c58: ffffffc00e7773f0
ffffff800c9f3cc8: ffffffc00e7773f0
ffffff800c9f3cf8: ffffffc00e7773f0
PID: 5857 TASK: ffffffc003234880 CPU: 2 COMMAND: "Binder:32060_E"//binder
ffffff80105cb478: ffffffc00e7773f0
[118543 652667693] [IN] PID: 5857 TASK: ffffffc003234880 CPU: 2 COMMAND: "Binder:32060_E"
PID: 19825 TASK: ffffffc017708000 CPU: 0 COMMAND: "UserFacingBlock"//
ffffff800f82bda0: ffffffc00e7773f0
ffffff800f82be38: ffffffc00e7773f0
ffffff800f82be48: ffffffc00e7773f0
PID: 19826 TASK: ffffffc017708e80 CPU: 1 COMMAND: "UserFacingBlock"//UN-ps
ffffff800f6dbda0: ffffffc00e7773f0
ffffff800f6dbe38: ffffffc00e7773f0
ffffff800f6dbe48: ffffffc00e7773f0
PID: 22126 TASK: ffffffc08845ab80 CPU: 0 COMMAND: "Blocking Thread"//UN-ps
ffffff800d223da0: ffffffc00e7773f0
ffffff800d223e38: ffffffc00e7773f0
ffffff800d223e48: ffffffc00e7773f0
PID: 22143 TASK: ffffffc042224880 CPU: 0 COMMAND: "UserFacingBlock"//UN-ps
ffffff801424bda0: ffffffc00e7773f0
ffffff801424be38: ffffffc00e7773f0
ffffff801424be48: ffffffc00e7773f0
PID: 22402 TASK: ffffffc08041d700 CPU: 1 COMMAND: "Blocking Thread"//UN-ps
ffffff800f9f3da0: ffffffc00e7773f0
ffffff800f9f3e38: ffffffc00e7773f0
ffffff800f9f3e48: ffffffc00e7773f0
PID: 22414 TASK: ffffffc002728000 CPU: 1 COMMAND: "Blocking Thread"//UN-ps
ffffff800c56bda0: ffffffc00e7773f0
ffffff800c56be38: ffffffc00e7773f0
ffffff800c56be48: ffffffc00e7773f0
PID: 22621 TASK: ffffffc05f8e8e80 CPU: 2 COMMAND: "Blocking Thread"//UN-ps
ffffff800f5cbda0: ffffffc00e7773f0
ffffff800f5cbe38: ffffffc00e7773f0
ffffff800f5cbe48: ffffffc00e7773f0
PID: 22697 TASK: ffffffc036bc0000 CPU: 7 COMMAND: "GAC_Executor[12"//UN-ps
ffffff800c94bda0: ffffffc00e7773f0
ffffff800c94be38: ffffffc00e7773f0
ffffff800c94be48: ffffffc00e7773f0
PID: 22698 TASK: ffffffc002729d00 CPU: 6 COMMAND: "GAC_Executor[13"//UN-ps
ffffff800dbdbda0: ffffffc00e7773f0
ffffff800dbdbe38: ffffffc00e7773f0
ffffff800dbdbe48: ffffffc00e7773f0
PID: 32073 TASK: ffffffc01b730e80 CPU: 6 COMMAND: "Jit thread pool"//UN-ps
ffffff800fc6bd70: ffffffc00e7773f0
ffffff800fc6be18: ffffffc00e7773f0
PID: 32074 TASK: ffffffc071fc8e80 CPU: 7 COMMAND: "HeapTaskDaemon"//UN-ps
ffffff800fcbbc30: ffffffc00e7773f0
ffffff800fcbbc38: ffffffc00e7773f0
ffffff800fcbbcd8: ffffffc00e7773f0
PID: 32078 TASK: ffffffc052b66580 CPU: 3 COMMAND: "Binder:32060_1" //binder
ffffff8010793478: ffffffc00e7773f0
ffffff8010793618: ffffffc00e7773f0
ffffff8010793678: ffffffc00e7773f0
[118543652560039] [IN] PID: 32078 TASK: ffffffc052b66580 CPU: 3 COMMAND: "Binder:32060_1"
PID: 32079 TASK: ffffffc0793fc880 CPU: 2 COMMAND: "Binder:32060_2" //binder
ffffff80107a3478: ffffffc00e7773f0
ffffff80107a3618: ffffffc00e7773f0
ffffff80107a3678: ffffffc00e7773f0
[118543652579578] [IN] PID: 32079 TASK: ffffffc0793fc880 CPU: 2 COMMAND: "Binder:32060_2"//
PID: 32080 TASK: ffffffc0137f0e80 CPU: 5 COMMAND: "Binder:32060_3"//binder
ffffff8010d2b6c8: ffffffc00e7773f0
ffffff8010d2b728: ffffffc00e7773f0
[118543 652 667 193] [IN] PID: 32080 TASK: ffffffc0137f0e80 CPU: 5 COMMAND: "Binder:32060_3"
PID: 32125 TASK: ffffffc0369f2b80 CPU: 0 COMMAND: "RenderThread"//UN-ps
ffffff80128f3d20: ffffffc00e7773f0
ffffff80128f3d90: ffffffc00e7773f0
ffffff80128f3dc8: ffffffc00e7773f0
PID: 32130 TASK: ffffffc00473ab80 CPU: 0 COMMAND: "mali-mem-purge"
ffffff8012723da0: ffffffc00e7773f0
ffffff8012723e38: ffffffc00e7773f0
ffffff8012723e48: ffffffc00e7773f0
PID: 32131 TASK: ffffffc0383e0e80 CPU: 4 COMMAND: "mali-utility-wo"
ffffff80126f3de8: ffffffc00e7773f0
[116172 885 652 160] [IN] PID: 32131 TASK: ffffffc0383e0e80 CPU: 4 COMMAND: "mali-utility-wo"
PID: 32132 TASK: ffffffc0383e0000 CPU: 0 COMMAND: "mali-utility-wo"
ffffff80127936e8: ffffffc00e7773f0
ffffff8012793798: ffffffc00e7773f0
ffffff8012793de8: ffffffc00e7773f0
[116172885680660] [IN] PID: 32132 TASK: ffffffc0383e0000 CPU: 0 COMMAND: "mali-utility-wo"
PID: 32133 TASK: ffffffc047a3e580 CPU: 7 COMMAND: "mali-utility-wo"
ffffff801279b6e8: ffffffc00e7773f0
ffffff801279b798: ffffffc00e7773f0
ffffff801279bde8: ffffffc00e7773f0
PID: 32134 TASK: ffffffc047a3d700 CPU: 7 COMMAND: "mali-utility-wo"
ffffff801283b6e8: ffffffc00e7773f0
ffffff801283b798: ffffffc00e7773f0
ffffff801283bde8: ffffffc00e7773f0
PID: 32136 TASK: ffffffc00f472b80 CPU: 7 COMMAND: "mali-utility-wo"
ffffff80128e36e8: ffffffc00e7773f0
ffffff80128e3798: ffffffc00e7773f0
PID: 32137 TASK: ffffffc08349d700 CPU: 0 COMMAND: "mali-utility-wo"
ffffff801294b6e8: ffffffc00e7773f0
ffffff801294b798: ffffffc00e7773f0
[116172885730891] [IN] PID: 32137 TASK: ffffffc08349d700 CPU: 0 COMMAND: "mali-utility-wo"
crash_arm64> list -s rwsem_waiter.task,type -h 0xffffff8012723dd8
ffffff8012723dd8
task = 0xffffffc00473ab80
type = RWSEM_WAITING_FOR_WRITE
ffffff800fc6bda8
task = 0xffffffc01b730e80
type = RWSEM_WAITING_FOR_WRITE
ffffff80128f3d58
task = 0xffffffc0369f2b80
type = RWSEM_WAITING_FOR_WRITE
ffffff800dbdbdd8
task = 0xffffffc002729d00
type = RWSEM_WAITING_FOR_WRITE
ffffff800c94bdd8
task = 0xffffffc036bc0000
type = RWSEM_WAITING_FOR_WRITE
ffffff800fb03c98
task = 0x0
type = RWSEM_WAITING_FOR_WRITE
list: invalid list entry: 0
static ssize_t get_mm_cmdline(struct mm_struct *mm, char __user *buf,
215 size_t count, loff_t *ppos)
216 {
217 unsigned long arg_start, arg_end, env_start, env_end;
218 unsigned long pos, len;
219 char *page;
220
221 /* Check if process spawned far enough to have cmdline. */
222 if (!mm->env_end)
223 return 0;
224
225 down_read(&mm->mmap_sem);
226 arg_start = mm->arg_start;
227 arg_end = mm->arg_end;
228 env_start = mm->env_start;
229 env_end = mm->env_end;
230 up_read(&mm->mmap_sem);
231
struct mm_struct { -- ffffffc00e777380
[0x70] struct rw_semaphore mmap_sem; -- ffffffc00e7773f0
[0x310] struct task_struct *owner; -- ffffffc001c38e80
crash_arm64> eval ffffffc00e777380+0x310
hexadecimal: ffffffc00e777690
decimal: 18446743799074354832 (-274635196784)
octal: 1777777774001635673220
binary: 1111111111111111111111111100000000001110011101110111011010010000
crash_arm64> rd ffffffc00e777690
ffffffc00e777690: ffffffc001c38e80 ........
crash_arm64> struct task_struct ffffffc001c38e80
pid = 32060,
comm = "earchbox:search",
目前来看watchdog在读取32060的proc节点时被阻塞了，该进程的mmap_sem被其他进程持有了，需要找出谁持有了该进程。
从search -t来看，最大的嫌疑犯是4830。
PID: 4830 TASK: ffffffc00fa8c880 CPU: 0 COMMAND: "Chrome_IOThread"
#0 [ffffff800a2037d0] __switch_to at ffffff80081f4ae0
#1 [ffffff800a203820] __schedule at ffffff8008e1518c
#2 [ffffff800a203880] schedule at ffffff8008e15528
#3 [ffffff800a203900] schedule_timeout at ffffff8008e193cc
#4 [ffffff800a203960] io_schedule_timeout at ffffff8008e15a74
#5 [ffffff800a203b50] balance_dirty_pages_ratelimited at ffffff800838bd24
#6 [ffffff800a203bc0] do_wp_page at ffffff80083c3ba0
#7 [ffffff800a203c10] handle_pte_fault at ffffff80083c4c90
#8 [ffffff800a203cf0] handle_mm_fault at ffffff80083c5b70
#9 [ffffff800a203d90] do_page_fault$2ad2e1b83c15cf95a0f028b354d101f0 at ffffff8008210510
#10 [ffffff800a203e80] do_mem_abort at ffffff80080815dc
#11 [ffffff800a203ff0] el0_da at ffffff8008084054
#7 [ffffff800a203c10] handle_pte_fault at ffffff80083c4c90
ffffff800a203c10: ffffff800a203cf0 ffffff80083c5b74
ffffff800a203c20: ffffffc071126ca8 ffffffc00e7773f0## rw_semaphore##
ffffff800a203c30: 0000000000000055 ffffffc00fa8c880
ffffff800a203c40: ffffffc0b995ced8 ffffffc071126ca8
ffffff800a203c50: 00000076ef2ce000 0000000000000055
ffffff800a203c60: ffffffc071126ca8 014200ca00000055
ffffff800a203c70: 0000000000000004 00000076ef2ce000
ffffff800a203c80: 0000000000000006 0000000000000000
ffffff800a203c90: ffffffc0b5455bc8 ffffffc0b995ced8
ffffff800a203ca0: 00600000ee987fd3 0000000000000000
ffffff800a203cb0: 0000000000000000 ffffffbf01ba61c0
ffffff800a203cc0: ffffffc01219a670 ffffffbf004866b0
ffffff800a203cd0: 0000000000000000 00000000000000fb
ffffff800a203ce0: 0060000000000fd3 4345862aad4c8f00
阅读源码发现
471 retry:
472 down_read(&mm->mmap_sem);
473 } else {
474 /*
475 * The above down_read_trylock() might have succeeded in
476 * which case we'll have missed the might_sleep() from
477 * down_read():
478 */
479 might_sleep();
480 }
481
482 vma = find_vma(mm, address);
483 if (unlikely(!vma))
484 return bad_area(regs, address);
485 if (likely(vma->vm_start <= address))
486 goto good_area;
487 if (unlikely(!(vma->vm_flags & VM_GROWSDOWN)))
488 return bad_area(regs, address);
489
490 /* The stack is being expanded, check if it's valid */
491 if (unlikely(bad_stack_expansion(regs, address, vma, store_update_sp)))
492 return bad_area(regs, address);
493
494 /* Try to expand it */
495 if (unlikely(expand_stack(vma, address)))
496 return bad_area(regs, address);
497
498 good_area:
499 if (unlikely(access_error(is_write, is_exec, vma)))
500 return bad_access(regs, address);
501
502 /*
503 * If for any reason at all we couldn't handle the fault,
504 * make sure we exit gracefully rather than endlessly redo
505 * the fault.
506 */
507 fault = handle_mm_fault(vma, address, flags);
508 major |= fault & VM_FAULT_MAJOR;
代码中确实存在down_read后，handle_mm_fault - handle_pte_fault - do_wp_page - balance_dirty_pages_ratelimited - schedule
当前UN进程里存在
ffffffc00e7773f0
ffffffc0144652c8
ffffffc004cf9ff0
ffffffc004cf91f0
ffffffc004cfdb70
ffffffc00e7773f0 ffffffc004cfdb70 和 ffffffc004cf91f0 ffffffc004cf91f0d都被balance_dirty_pages_ratelimited出去。
ffffffc0144652c8被15577 block balance_dirty_pages_ratelimited - schedule
#3 [ffffff8011a13c90] rwsem_down_write_failed at ffffff8008e18a90
ffffff8011a13c90: ffffff8011a13ce0 ffffff8008e17dc8
ffffff8011a13ca0: 00000000000001b3 0000000000000000
ffffff8011a13cb0: ffffffc0a533c880 0000000000000000
ffffff8011a13cc0: ffffffc0144652c8 ffffff8011a13db8
ffffff8011a13cd0: ffffffc014465220 ffffffc0144652c8
crash_arm64> struct rw_semaphore ffffffc0144652c8 -x
struct rw_semaphore {
count = {
counter = 0xfffffffe00000001 owner = 0xffffffc08573d700
},
wait_list = {
next = 0xffffff8011a13c68,
prev = 0xffffff8011a13c68
},
wait_lock = {
raw_lock = {
owner = 0x1,
next = 0x1
}
},
osq = {
tail = {
counter = 0x0
}
},
owner = 0xffffffc08573d700
}
crash_arm64> task_struct 0xffffffc08573d700 | grep mm
comm = "Thread-18\000nitor",
pid = 15577,
crash_arm64> bt 15577
PID: 15577 TASK: ffffffc08573d700 CPU: 7 COMMAND: "Thread-18"
#0 [ffffff8013c83840] __switch_to at ffffff80081f4ae0
#1 [ffffff8013c83890] __schedule at ffffff8008e1518c
#2 [ffffff8013c838f0] schedule at ffffff8008e15528
#3 [ffffff8013c83970] schedule_timeout at ffffff8008e193cc
#4 [ffffff8013c839d0] io_schedule_timeout at ffffff8008e15a74
#5 [ffffff8013c83bc0] balance_dirty_pages_ratelimited at ffffff800838bd24
#6 [ffffff8013c83c50] generic_perform_write at ffffff800837e328
#7 [ffffff8013c83cb0] __generic_file_write_iter at ffffff800837e468
#8 [ffffff8013c83d40] f2fs_file_write_iter$717f3c7ebb2509b4b38e79ecff9af67f at ffffff80085a100c
#9 [ffffff8013c83e20] __vfs_write at ffffff80083ff6b0
#10 [ffffff8013c83e40] vfs_write at ffffff80083ff9f0
#11 [ffffff8013c83e90] sys_write at ffffff80083ffc34
#12 [ffffff8013c83ff0] el0_svc_naked at ffffff80080844bc
PC: 0000007a4f1388c8 LR: 00000077b40ee834 SP: 0000007718caf4e0
X29: 0000007718caf510 X28: 0000000000000001 X27: 0000000000000000
X26: 0000007718cb2000 X25: 0000007718cb2000 X24: b4000077dc835030
X23: b40000792c815a08 X22: 0000000000000037 X21: 000000000000004a
X20: 0000007718caf588 X19: b40000787c7e1370 X18: 0000007717dd6000
X17: 0000007a4f1388c0 X16: 0000007a4f156ef8 X15: 0000000000000001
X14: 00000077bc4902f8 X13: 0000000000000000 X12: 00000000c8dbf66e
X11: 00000000dc835050 X10: 0000000000000002 X9: 0000000000000001
X8: 0000000000000040 X7: 7f7f7f7f7f7f7f7f X6: 284801ff3a445328
X5: 0000000000000049 X4: ffffffffffffffff X3: ffffffffffffffff
X2: 0000000000000037 X1: b40000793c7e0090 X0: 000000000000004a
ORIG_X0: 000000000000004a SYSCALLNO: 40 PSTATE: 00001000
4008
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
