Linux内核中KCSAN 数据竞争检测-CSDN博客

CONFIG_KCSAN = y

CONFIG_KCSAN_ASSUME_PLAIN_WRITES_ATOMIC=n

CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY=n

CONFIG_KCSAN_INTERRUPT_WATCHER=y

CONFIG_DEBUG_KERNEL=y

2、内核提供的测试用例

源码位置：kernel/kcsan/kcsan-test.c

编译文件：kcsan-test.ko

3、检测输出

选择其中两类输出

[  426.071584] ==================================================================
[  426.071584] BUG: KCSAN: data-race in test_kernel_read [kcsan_test] / test_kernel_write [kcsan_test]
[  426.071584] 
[  426.071584] read to 0xffffffffc0203c60 of 8 bytes by task 307 on cpu 2:
[  426.071584]  test_kernel_read+0x19/0x30 [kcsan_test]
[  426.101893]  access_thread+0x43/0xb0 [kcsan_test]
[  426.101893]  kthread+0x183/0x1b0
[  426.101893]  ret_from_fork+0x22/0x30
[  426.101893] 
[  426.101893] write to 0xffffffffc0203c60 of 8 bytes by task 306 on cpu 1:
[  426.101893]  test_kernel_write+0x22/0x40 [kcsan_test]
[  426.101893]  access_thread+0x43/0xb0 [kcsan_test]
[  426.101893]  kthread+0x183/0x1b0
[  426.101893]  ret_from_fork+0x22/0x30
[  426.101893] 
[  426.101893] Reported by Kernel Concurrency Sanitizer on:
[  426.101893] CPU: 1 PID: 306 Comm: access_thread Kdump: loaded Tainted: G                 N 6.0.0-rc6 #5
[  426.101893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[  426.101893] ==================================================================
[  427.253735] ==================================================================
[  427.253735] BUG: KCSAN: data-race in test_kernel_read+0x19/0x30 [kcsan_test]
[  427.253735] 
[  427.253735] race at unknown origin, with read to 0xffffffffc0203c60 of 8 bytes by task 307 on cpu 2:
[  427.274398]  test_kernel_read+0x19/0x30 [kcsan_test]
[  427.274398]  access_thread+0x43/0xb0 [kcsan_test]
[  427.274398]  kthread+0x183/0x1b0
[  427.274398]  ret_from_fork+0x22/0x30
[  427.274398] 
[  427.274398] value changed: 0x000000000017b4a9 -> 0x000000000017b520
[  427.274398] 
[  427.274398] Reported by Kernel Concurrency Sanitizer on:
[  427.274398] CPU: 2 PID: 307 Comm: access_thread Kdump: loaded Tainted: G                 N 6.0.0-rc6 #5
[  427.274398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014
[  427.274398] ==================================================================
[  427.732807] test_basic-torture: Stopping reader_thread task

4、检测解析

BUG: KCSAN: data-race

read/write [(marked)] to <kernel-virt-addr> of <n> bytes by task <PID> on cpu <CPU#>

所涉及的是读写操作

“value changed” 如 value changed: 0x000000000017b4a9 -> 0x000000000017b520

BUG: KCSAN: data-race in test_kernel_read+0x19/0x30

三、锁的注意事项

1、通过debugfs查看

/sys/kernel/debug/kcsan 查看kcsan的参数

# cat /sys/kernel/debug/kcsan 
enabled: 1
used_watchpoints: 0
setup_watchpoints: 177185
data_races: 12
assert_failures: 0
no_capacity: 0
report_races: 5
races_unknown_origin: 0
unencodable_accesses: 0
encoding_false_positives: 0

blacklisted functions: none

2、加锁注意事项

容易导致锁缺陷的情况

        尝试两次加锁
        尝试释放没有上锁的锁
        退出而不释放所

在自旋锁中申请内存，应该使用GFP_ATOMIC标志，而不是GFP_KERNEL
加锁的原则：仅在绝对必要时禁用中断，然后尽可能短的时间

3、Local Locks

新的同步原语，在5.8内核中添加

本地锁是抢占和中断启用/禁用原语的包装器。特别是对于调试内核来说，使用lockdep和静态分析是发现bug的关键。

在非实时系统上，获取本地锁只会映射到禁用抢占（以及可能的中断）。在实时系统上，本地锁实际上是休眠的自旋锁；它们不会禁用抢占或中断。这足以对受保护资源的访问，而不会增加整个系统的延迟。

本地锁定操作按以下方式映射到抢占和中断禁用

local_lock(&llock) preempt_disable()
local_unlock(&llock) preempt_enable()
local_lock_irq(&llock) local_irq_disable()
local_unlock_irq(&llock) local_irq_enable()
local_lock_save(&llock) local_irq_save()
local_unlock_restore(&llock) local_irq_restore()

参考 Concurrency bugs should fear the big bad data-race detector (part 1) [LWN.net]

四、锁的错误使用

1、不正确的自旋锁用法

内核源码驱动位置： drivers/tty/tty_jobctrl.c:tiocspgrp()

使用了错误的锁

kernel version 5.16 才修复

2、阻塞和无法获取引用

1）、在临界资源中不要睡眠

rcu_read_lock(); // begin RCU read critical section
[...]
msleep(10); /* bug! *sleep() helpers all block; if
you must, use the *delay() helpers instead (they're
non-blocking) */
rcu_read_unlock(); // end RCU read critical section

2）、使用内核全局数据结构时，使用后要释放。因其内部有申请与释放的操作

get_task_struct();
/* ... use it ... */
put_task_struct();

get_file(file);
/* ... use the file ... */
fput(file);

原文 My First Kernel Module: A Debugging Nightmare

3、长时间的延迟（数据处理）

spin_lock_irq[save](&mylock[, flags]); /* disables interrupts
*/
// time t1
/* ... do the work ... */
// time t2
spin_unlock_irq[restore](&mylock[, flags]); /* enables
interrupts */

在处理数据消耗的时间自旋锁 t2-t1

t2-t1 非常小（几微秒或者一位数的毫秒），一般情况下运行正常；

t2-t1 比较大（几十毫秒或者更长），这种情况不正常，会引起各种延迟问题，甚至是livelock

3.1 阿里的一个锁的问题

原文 Network Jitter: An In-Depth Case Study - Alibaba Cloud Community

网络抖动是由于系统中的延迟很长而导致的。工程师发现，slab统计信息查找调用spin_lock_irq（）的代码时引入的延迟。正如我们所知，它在内部禁用了硬件中断，因为这段代码在循环中运行了很长一段时间。O（n）时间复杂性

这是一个非常典型的问题：迭代大于预期的列表，导致意外延迟。

因为spin_lock_irq[save]（）API同时禁用硬件irq和内核抢占，所以我们必须尽快重新启用它们 spin_unlock_ir q[restore]（）。保持自旋锁所花费的时间超过几十毫秒就被认为太长了。

锁时间的测量方法

1、eBPF

criticalstat：在内核中查找原子临界区的时间

# ./criticalstat -h
usage: criticalstat [-h] [-p] [-i] [-d DURATION]

Trace long critical sections

optional arguments:
  -h, --help            Show this help message and exit
  -p, --preemptoff      Find long sections where preemption was off
  -i, --irqoff          Find long sections where IRQ was off
  -d DURATION, --duration DURATION
                        Duration in uS (microseconds) below which we filter

examples:
    ./criticalstat          	# run with default options: irq off for more than 100 uS
    ./criticalstat -p       	# find sections with preemption disabled for more than 100 uS
    ./criticalstat -d 500   	# find sections with IRQs disabled for more than 500 uS
    ./criticalstat -p -d 500	# find sections with preemption disabled for more than 500 uS