背景介绍
我们的客户在利用我们提供的SDK参考方案开发相关产品时，在产品方案上进行一些基础老化测试时，极低概率出现kernel随机panic问题，由于场景复杂，无法单独针对特定模块或功能进行拆解来进行实验排查，只能基于已有的log和复现抓kdump进行相关分析和定位。本文记录相关分析过程，以作后续参考和学习！
问题log分析
首次抓到现场的关键log如下所示：
[ 153.081557] Unable to handle kernel paging request at virtual address 1800180018000
[ 153.089255] Mem abort info:
[ 153.092058] Exception class = DABT (current EL), IL = 32 bits
[ 153.097997] SET = 0, FnV = 0
[ 153.101068] EA = 0, S1PTW = 0
[ 153.104208] Data abort info:
[ 153.107098] ISV = 0, ISS = 0x00000004
[ 153.110938] CM = 0, WnR = 0
[ 153.113916] [0001800180018000] address between user and kernel address ranges
[ 153.121058] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[ 153.126630] Modules linked in: bcmdhd hci_uart bluetooth ecdh_generic crypto_engine pvrsrvkm
[ 153.135082] CPU: 3 PID: 2723 Comm: MessageThread Tainted: G O 4.14.61-00009-gc2eae318e529-dirty #14
[ 153.151001] task: ffff800178125400 task.stack: ffff00000ad78000
[ 153.156928] PC is at test_and_set_bit+0x18/0x38
[ 153.161461] LR is at zs_page_migrate+0x104/0x560
[ 153.166077] pc : [<ffff000008cf69a8>] lr : [<ffff00000828c814>] pstate: 40400145
[ 153.173471] sp : ffff00000ad7b610
[ 153.176784] x29: ffff00000ad7b6a0 x28: ffff8000eb502060
[ 153.182096] x27: 00000000000008e0 x26: ffff80005d715000
[ 153.187409] x25: ffff800178125400 x24: ffff8000eb502088
[ 153.192721] x23: ffff7e000175c550 x22: ffff7e0004615680
[ 153.198033] x21: ffff7e000175c540 x20: ffff800158629b00
[ 153.203345] x19: 00000000000008e0 x18: 0000000000000000
[ 153.208657] x17: 0000000000000001 x16: ffff80017ff16b00
[ 153.213969] x15: 0001182270000000 x14: ffff0000091d2018
[ 153.219280] x13: 00000000000000c1 x12: 0000000000000000
[ 153.224592] x11: 0000000000000140 x10: 0000000000000000
[ 153.229904] x9 : 0000000000000005 x8 : 8001800180018001
[ 153.235215] x7 : 8000800080008000 x6 : ffff8000d8228000
[ 153.240527] x5 : 0000000000000000 x4 : 0000000000000001
[ 153.245839] x3 : 0000000000000000 x2 : 0000000000000001
[ 153.251150] x1 : 8001800180018000 x0 : 0000000000000000
[ 153.256465]
[ 153.256465] X6: 0xffff8000d8227f80:
[ 153.261428] 7f80 331c2a00 01200207 580e2a00 1120020c 80000000 80008000 80008000 80008000
[ 153.269614] 7fa0 80008000 80008000 80008000 80008000 80008000 80008000 80008000 80008000
[ 153.277797] 7fc0 80008000 80008000 80008000 80008000 80008000 80008000 80008000 80008000
[ 153.285982] 7fe0 80008000 80008000 80008000 80008000 80008000 80008000 80008000 80008000
[ 153.294166] 8000 e2c56740 0000e7c5 f804ae40 0000e7c5 0000000f 00000000 00000000 00000000
[ 153.302351] 8020 00000000 00000000 00000000 408f4000 00000008 00000000 7375732a 646e6570
[ 153.310535] 8040 6c6c6120 73696820 72676f74 00006d61 00000040 00000000 00000008 00000000
[ 153.318719] 8060 00000010 00000000 e3e91c00 0000e7c5 e3e91c84 0000e7c5 e3e91d00 0000e7c5
[ 153.326906]
[ 153.326906] X16: 0xffff80017ff16a80:
[ 153.331956] 6a80 00000001 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.340139] 6aa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.348323] 6ac0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.356507] 6ae0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.364690] 6b00 0481b7e0 ffff7e00 03801ae0 ffff7e00 0357f8e0 ffff7e00 0357f8e0 ffff7e00
[ 153.372875] 6b20 03b9fc60 ffff7e00 037034a0 ffff7e00 7ff16b30 ffff8001 7ff16b30 ffff8001
[ 153.381059] 6b40 7ff16b40 ffff8001 7ff16b40 ffff8001 7ff16b50 ffff8001 7ff16b50 ffff8001
[ 153.389243] 6b60 0000003c 00000000 03801a20 ffff7e00 03804520 ffff7e00 041091a0 ffff7e00
[ 153.397428]
[ 153.397428] X20: 0xffff800158629a80:
[ 153.402477] 9a80 000008ab 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.410660] 9aa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.418843] 9ac0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.425337] saf775x saf775d_ioctl The process is "Binder:3409_2" (pid 3456)
[ 153.425340] radio cmd:SAF775D_RADIO_GETLEVEL
[ 153.425928] lllevel = -7
[ 153.440955] 9ae0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.449138] 9b00 054c054a 00000000 58629b08 ffff8001 58629b08 ffff8001 20753e28 ffff8001
[ 153.457322] 9b20 234698e8 ffff8000 01499a68 ffff8001 cd9bb408 ffff8000 236ba9a8 ffff8000
[ 153.465507] 9b40 4d7b3e58 ffff8000 000006d0 00000007 00000003 0000006b 00000000 00000000
[ 153.473691] 9b60 00000003 00000000 00000003 00000000 00000092 00000000 0000042f 00000000
[ 153.481877]
[ 153.481877] X24: 0xffff8000eb502008:
[ 153.486927] 2008 00000007 00000000 0175c380 ffff7e00 eb502018 ffff8000 eb502018 ffff8000
[ 153.495111] 2028 00000000 00000000 001609fb 00000002 ffffffff 00000000 0175c480 ffff7e00
[ 153.503296] 2048 eb502048 ffff8000 eb502048 ffff8000 00000000 00000000 001619af 00000007
[ 153.511480] 2068 00000007 00000000 0175c4c0 ffff7e00 eb502078 ffff8000 eb502078 ffff8000
[ 153.519663] 2088 80000000 00000000 001601fb 00000002 ffffffff 00000000 036089c0 ffff7e00
[ 153.527847] 20a8 eb5020d8 ffff8000 31ac4be8 ffff8000 00000000 00000000 001601fb 00000002
[ 153.536031] 20c8 ffffffff 00000000 03608300 ffff7e00 f272daf8 ffff8000 eb5020a8 ffff8000
[ 153.544215] 20e8 00000000 00000000 001603fb 00000001 ffffffff 00000000 048ce700 ffff7e00
[ 153.552399]
[ 153.552399] X25: 0xffff800178125380:
[ 153.557449] 5380 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.565632] 53a0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.573816] 53c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.582000] 53e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.590183] 5400 0000002a 00000000 ffffffff ffffffff 00000006 00000000 00000000 00000000
[ 153.598367] 5420 0ad78000 ffff0000 00000003 00c04040 00000000 00000000 00000000 00000000
[ 153.606552] 5440 00000001 00000003 0000001a 00000000 ffff7081 00000000 77402a00 ffff8001
[ 153.614735] 5460 00000003 00000001 00000070 00000070 00000070 00000000 08d40d68 ffff0000
[ 153.622920]
[ 153.622920] X26: 0xffff80005d714f80:
[ 153.627970] 4f80 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.636154] 4fa0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.644338] 4fc0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.652523] 4fe0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.660707] 5000 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.668890] 5020 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.677074] 5040 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.685259] 5060 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.693443]
[ 153.693443] X28: 0xffff8000eb501fe0:
[ 153.698492] 1fe0 eb5e5018 ffff8000 00000000 00000000 00000000 00000000 00000000 00000000
[ 153.706676] 2000 00162243 00000007 00000007 00000000 0175c380 ffff7e00 eb502018 ffff8000
[ 153.714860] 2020 eb502018 ffff8000 00000000 00000000 001609fb 00000002 ffffffff 00000000
[ 153.723044] 2040 0175c480 ffff7e00 eb502048 ffff8000 eb502048 ffff8000 00000000 00000000
[ 153.731228] 2060 001619af 00000007 00000007 00000000 0175c4c0 ffff7e00 eb502078 ffff8000
[ 153.739412] 2080 eb502078 ffff8000 80000000 00000000 001601fb 00000002 ffffffff 00000000
[ 153.747596] 20a0 036089c0 ffff7e00 eb5020d8 ffff8000 31ac4be8 ffff8000 00000000 00000000
[ 153.755780] 20c0 001601fb 00000002 ffffffff 00000000 03608300 ffff7e00 f272daf8 ffff8000
[ 153.763965]
[ 153.765455] Process MessageThread (pid: 2723, stack limit = 0xffff00000ad78000)
[ 153.772762] Call trace:
[ 153.775208] Exception stack(0xffff00000ad7b4d0 to 0xffff00000ad7b610)
[ 153.781649] b4c0: 0000000000000000 8001800180018000
[ 153.789479] b4e0: 0000000000000001 0000000000000000 0000000000000001 0000000000000000
[ 153.797309] b500: ffff8000d8228000 8000800080008000 8001800180018001 0000000000000005
[ 153.805139] b520: 0000000000000000 0000000000000140 0000000000000000 00000000000000c1
[ 153.812969] b540: ffff0000091d2018 0001182270000000 ffff80017ff16b00 0000000000000001
[ 153.820798] b560: 0000000000000000 00000000000008e0 ffff800158629b00 ffff7e000175c540
[ 153.828629] b580: ffff7e0004615680 ffff7e000175c550 ffff8000eb502088 ffff800178125400
[ 153.836459] b5a0: ffff80005d715000 00000000000008e0 ffff8000eb502060 ffff00000ad7b6a0
[ 153.844289] b5c0: ffff00000828c814 ffff00000ad7b610 ffff000008cf69a8 0000000040400145
[ 153.852118] b5e0: ffff7e000175c300 0000000000000001 0000ffffffffffff ffff00000ad7b6e8
[ 153.859947] b600: ffff00000ad7b6a0 ffff000008cf69a8
[ 153.864825] [<ffff000008cf69a8>] test_and_set_bit+0x18/0x38
[ 153.870399] [<ffff0000082770b4>] move_to_new_page+0x2bc/0x2e0
[ 153.876145] [<ffff0000082755ec>] migrate_pages+0x62c/0x928
[ 153.881632] [<ffff000008208400>] alloc_contig_range+0x194/0x4ac
[ 153.887553] [<ffff00000828e4a4>] cma_alloc+0x144/0x2f4
[ 153.892693] [<ffff0000086d5c04>] dma_alloc_from_contiguous+0x2c/0x34
[ 153.899048] [<ffff00000809ea08>] __dma_alloc+0x150/0x298
[ 153.904362] [<ffff00000893253c>] vb2_dc_alloc+0x104/0x164
[ 153.909759] [<ffff00000892c798>] __vb2_queue_alloc+0x184/0x494
[ 153.915592] [<ffff00000892bc6c>] vb2_core_reqbufs+0x264/0x42c
[ 153.921337] [<ffff000008931294>] vb2_ioctl_reqbufs+0x6c/0x94
[ 153.926997] [<ffff0000089174ec>] v4l_reqbufs+0x48/0x58
[ 153.932135] [<ffff000008914f1c>] __video_do_ioctl+0x130/0x26c
[ 153.937880] [<ffff000008914b00>] video_usercopy+0x2f0/0x5c0
[ 153.943453] [<ffff000008914de4>] video_ioctl2+0x14/0x1c
[ 153.948678] [<ffff0000089142e4>] v4l2_ioctl+0x9c/0xc4
[ 153.953730] [<ffff0000082a913c>] do_vfs_ioctl+0x554/0x810
[ 153.959129] [<ffff0000082a9564>] SyS_ioctl+0x88/0x94
[ 153.964092] Exception stack(0xffff00000ad7bec0 to 0xffff00000ad7c000)
[ 153.970532] bec0: 0000000000000009 00000000c0145608 0000f168e12bcab8 0000000000000003
[ 153.978363] bee0: 0000000000000000 0000000000000018 fefff067e02aceff 7f7fff7fff7fff7f
[ 153.986193] bf00: 000000000000001d 0000f168e12bc9b8 0000f168e12bc980 0000f168e12bc9b8
[ 153.994023] bf20: 0000f168e12bca00 ffffffffffffffff 0000000000000004 ffffffffffffffff
[ 154.001853] bf40: 0000f168e1328f18 0000f168e2890308 0000f168df8c8000 0000f168e12bd020
[ 154.009682] bf60: 0000f168e2051000 0000000000000000 0000f168e12bca70 0000f168e12bd020
[ 154.017512] bf80: 0000f168e12bcd50 0000f168e12bcd50 0000f168e12bd020 0000f168e3585020
[ 154.025342] bfa0: 0000f168e1313708 0000f168e12bc9f0 0000f168e2890390 0000f168e12bc900
[ 154.033172] bfc0: 0000f168e28d2888 00000000a0000000 0000000000000009 000000000000001d
[ 154.041002] bfe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 154.048832] [<ffff000008083ac0>] el0_svc_naked+0x34/0x38
[ 154.054144] Code: d2800022 8b400c21 f9800031 9ac32044 (c85f7c22)
[ 154.060244] SMP: stopping secondary CPUs
[ 154.064172] ---[ end trace d3ae0601ebdf78fc ]---
[ 154.071764] Kernel panic - not syncing: Fatal exception
[ 154.076996] SMP: stopping secondary CPUs
[ 155.080920] SMP: failed to stop secondary CPUs 0-3
结合现场和对应kernel版本的vmlinux分析，可以推算出在test_and_set_bit()函数里出现了访问非法地址的行为，触发了data abort异常，对应汇编信息如下：
ffff000008cf6990 <test_and_set_bit>:
ffff000008cf6990: 12001403 and w3, w0, #0x3f //x3 = x0 & 0x3f
ffff000008cf6994: 4a030000 eor w0, w0, w3 //x0 = x0^x3
ffff000008cf6998: d2800022 mov x2, #0x1 // #1
ffff000008cf699c: 8b400c21 add x1, x1, x0, lsr #3 //x1 = x1 + (x0 >> 3)
ffff000008cf69a0: f9800031 prfm pstl1strm, [x1]
ffff000008cf69a4: 9ac32044 lsl x4, x2, x3
ffff000008cf69a8: c85f7c22 ldxr x2, [x1] //panic, x1=0x8001800180018000, x0=0
ffff000008cf69ac: 9ac32440 lsr x0, x2, x3
ffff000008cf69b0: aa040042 orr x2, x2, x4
ffff000008cf69b4: c805fc22 stlxr w5, x2, [x1]
ffff000008cf69b8: 35ffff85 cbnz w5, ffff000008cf69a8 <test_and_set_bit+0x18>
ffff000008cf69bc: d5033bbf dmb ish
ffff000008cf69c0: 92400000 and x0, x0, #0x1
ffff000008cf69c4: d65f03c0 ret
函数的调用路径及实现如下：
zs_page_migrate(struct address_space *mapping, struct page *newpage, struct page *page, enum migrate_mode mode)
trypin_tag(unsigned long handle)
bit_spin_trylock(int bitnum, unsigned long *addr)
static inline int trypin_tag(unsigned long handle)
{
return bit_spin_trylock(HANDLE_PIN_BIT, (unsigned long *)handle);
}
...
static int zs_page_migrate(struct address_space *mapping, struct page *newpage,
struct page *page, enum migrate_mode mode)
{
struct zs_pool *pool;
struct size_class *class;
int class_idx;
enum fullness_group fullness;
struct zspage *zspage;
struct page *dummy;
void *s_addr, *d_addr, *addr;
int offset, pos;
unsigned long handle, head;
unsigned long old_obj, new_obj;
unsigned int obj_idx;
int ret = -EAGAIN;
/*
* We cannot support the _NO_COPY case here, because copy needs to
* happen under the zs lock, which does not work with
* MIGRATE_SYNC_NO_COPY workflow.
*/
if (mode == MIGRATE_SYNC_NO_COPY)
return -EINVAL;
VM_BUG_ON_PAGE(!PageMovable(page), page);
VM_BUG_ON_PAGE(!PageIsolated(page), page);
zspage = get_zspage(page);
/* Concurrent compactor cannot migrate any subpage in zspage */
migrate_write_lock(zspage);
get_zspage_mapping(zspage, &class_idx, &fullness);
pool = mapping->private_data;
class = pool->size_class[class_idx];
offset = get_first_obj_offset(page);
spin_lock(&class->lock);
if (!get_zspage_inuse(zspage)) {
/*
* Set "offset" to end of the page so that every loops
* skips unnecessary object scanning.
*/
offset = PAGE_SIZE;
}
pos = offset;
s_addr = kmap_atomic(page);
while (pos < PAGE_SIZE) {
head = obj_to_head(page, s_addr + pos);
if (head & OBJ_ALLOCATED_TAG) {
handle = head & ~OBJ_ALLOCATED_TAG;
if (!trypin_tag(handle))
goto unpin_objects;
}
pos += class->size;
}
/*
* Here, any user cannot access all objects in the zspage so let's move.
*/
d_addr = kmap_atomic(newpage);
memcpy(d_addr, s_addr, PAGE_SIZE);
kunmap_atomic(d_addr);
for (addr = s_addr + offset; addr < s_addr + pos;
addr += class->size) {
head = obj_to_head(page, addr);
if (head & OBJ_ALLOCATED_TAG) {
handle = head & ~OBJ_ALLOCATED_TAG;
if (!testpin_tag(handle))
BUG();
old_obj = handle_to_obj(handle);
obj_to_location(old_obj, &dummy, &obj_idx);
new_obj = (unsigned long)location_to_obj(newpage,
obj_idx);
new_obj |= BIT(HANDLE_PIN_BIT);
record_obj(handle, new_obj);
}
}
replace_sub_page(class, zspage, newpage, page);
get_page(newpage);
dec_zspage_isolation(zspage);
/*
* Page migration is done so let's putback isolated zspage to
* the list if @page is final isolated subpage in the zspage.
*/
if (!is_zspage_isolated(zspage)) {
/*
* We cannot race with zs_destroy_pool() here because we wait
* for isolation to hit zero before we start destroying.
* Also, we ensure that everyone can see pool->destroying before
* we start waiting.
*/
putback_zspage_deferred(pool, class, zspage);
zs_pool_dec_isolated(pool);
}
if (page_zone(newpage) != page_zone(page)) {
dec_zone_page_state(page, NR_ZSPAGES);
inc_zone_page_state(newpage, NR_ZSPAGES);
}
reset_page(page);
put_page(page);
page = newpage;
ret = MIGRATEPAGE_SUCCESS;
unpin_objects:
for (addr = s_addr + offset; addr < s_addr + pos;
addr += class->size) {
head = obj_to_head(page, addr);
if (head & OBJ_ALLOCATED_TAG) {
handle = head & ~OBJ_ALLOCATED_TAG;
if (!testpin_tag(handle))
BUG();
unpin_tag(handle);
}
}
kunmap_atomic(s_addr);
spin_unlock(&class->lock);
migrate_write_unlock(zspage);
return ret;
}
结合源码分析，可以确认是handler这个变量出现问题了，从现场寄存器信息来看，x1=8001800180018000，与panic时log提示的现场一致
分析上下文，因为在zs_page_migrate()函数中直接调用了test_and_set_bit()，所以通过zs_page_migrate()的汇编代码看看是否可以获取handler的值，从汇编信息看，zs_page_migrate()中调用了test_and_set_bit()的汇编如下所示：
...
ffff00000828c7e0: 910042b7 add x23, x21, #0x10
ffff00000828c7e4: f94002a8 ldr x8, [x21]
ffff00000828c7e8: 8b1b0349 add x9, x26, x27
ffff00000828c7ec: f277011f tst x8, #0x200
ffff00000828c7f0: 9a970128 csel x8, x9, x23, eq // eq = none
ffff00000828c7f4: f9400108 ldr x8, [x8]
ffff00000828c7f8: 36000108 tbz w8, #0, ffff00000828c818 <zs_page_migrate+0x108>
ffff00000828c7fc: b9401329 ldr w9, [x25, #16]
ffff00000828c800: 927ff901 and x1, x8, #0xfffffffffffffffe
ffff00000828c804: 2a1f03e0 mov w0, wzr
ffff00000828c808: 11000529 add w9, w9, #0x1
ffff00000828c80c: b9001329 str w9, [x25, #16]
ffff00000828c810: 9429a860 bl ffff000008cf6990 <test_and_set_bit>
ffff00000828c814: 350009c0 cbnz w0, ffff00000828c94c <zs_page_migrate+0x23c>
ffff00000828c818: b9404a88 ldr w8, [x20, #72]
ffff00000828c81c: 0b130113 add w19, w8, w19
ffff00000828c820: 7140067f cmp w19, #0x1, lsl #12
ffff00000828c824: 93407e7b sxtw x27, w19
ffff00000828c828: 54fffde3 b.cc ffff00000828c7e4 <zs_page_migrate+0xd4>
...
因为test_and_set_bit()只有两个参数，x0和x1，从汇编分析，x0是0，x1 = x8 & 0xfffffffffffffffe, x8 = 0x8001800180018001， 所以x1 = 0x8001800180018000，因此，出现问题时，在zs_page_migrate()取到的x8的值就有问题，且x8表示head这个变量，0xfffffffffffffffe刚好是OBJ_ALLOCATED_TAG这个宏的取反，所有说明获取到的head变量已经出现不正确了，继续分析这个head变量的获取，源码如下:
static unsigned long obj_to_head(struct page *page, void *obj)
{
if (unlikely(PageHugeObject(page))) {
VM_BUG_ON_PAGE(!is_first_page(page), page);
return page->index;
} else
return *(unsigned long *)obj;
}
结合源码，在zs_page_migrate()函数中，通过obj_to_head()函数返回到对应的head，结合汇编分析，可以看出x23表示page->index，x9是obj，obj参数是在zs_page_migrate()计算出来的，该函数返回page->index或者返回obj这个指针指向的内存数据。
结合源码，可以推算出，x26表示s_addr这个变量，x26=0xffff80005d715000, x27表示pos这个变量，x27=0x00000000000008e0，因此obj这个指针的值是0xffff80005d7158e0，page这个指针的值是page = 0xffff7e000175c540，而handler就是从这两块内存地址获取到的。
从panic现场打印的数据信息看，head的值大概率应该是从x26这块内存获取到的，就是返回了obj这个指针指向的内存数据，因为现场没有打印出page这个指针指向的内存数据，只能从数据上进行猜测，但从x26这块内存数据看，这一块内存的数据都被填充成0x80018001，大概率是dma设备发生了内存踩踏导致，需要分析数据规律来进一步定位。
[ 153.622920] X26: 0xffff80005d714f80:
[ 153.627970] 4f80 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.636154] 4fa0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.644338] 4fc0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.652523] 4fe0 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.660707] 5000 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.668890] 5020 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.677074] 5040 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
[ 153.685259] 5060 80018001 80018001 80018001 80018001 80018001 80018001 80018001 80018001
3696
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
