Kernel Address Sanitizer (KASAN)
helps kernel developers and testers find runtime memory-related bugs, such as
out-of-bound read or write operations, and use-after-free issues. While KASAN
isn't enabled on production builds due to its runtime performance
penalties and memory usage increment, it is still a valuable tool for testing
debug builds.
When used with another runtime tool called Kernel Coverage (KCOV), KASAN-sanitized and
KCOV-instrumented code helps developers and testers to detect runtime memory
errors and obtain code coverage information. In the scenario of kernel fuzz
testing, e.g. via syzkaller,
KASAN helps to determine the root cause of crashes, and KCOV provides code
coverage information to the fuzzing engine to help in test-case or corpus
deduplication.
This page does not discuss the inner workings or mechanics of KASAN. Rather, it
serves as a guide to build and modify the Android Open Soure Project (AOSP) and
Pixel's kernel source to boot with KASAN and KCOV turned on.
Setting up your build
environment
Follow the steps in the Downloading and
Building section to set up your build environment.
Building AOSP
Download the Android source code. For the
purpose of building KASAN images, choose a stable build that is not in active
development. Often, the latest available release/stable branch is a good choice.
More information about build and branch can be found at Source
Code Tags and Builds.
After you have successfully checked out the source code, download the necessary
device blobs that correspond to the device and branch you are using from Driver Binaries for
Nexus and Pixel Devices. Download both the vendor image and the set of
binaries from the System-on-Chip (SOC) manufacturer. Then, unarchive the
downloaded tarballs, run the scripts they contain, and accept the licenses.
Tip: Double check that you have the right
version of JDK installed on your system before proceeding further.
Then clean up, set up your build environment, and choose your build target,
following the steps in
Preparing to Build.
To establish a working base, make your first build without modifications:
make -j48
Flash your build result to a test device (for example, marlin) and let it boot:
cd out/target/product/marlin
ANDROID_PRODUCT_OUT=`pwd` fastboot flashall -w
After booting to the homescreen, you might see a pop-up that says:
There's an internal problem with your device. Contact your manufacturer
for details. This pop-up likely means that the build fingerprint of your
vendor and your system partition do not match. Because this build is just for
development and testing, and not a release build, just ignore it.
Building the kernel
To build the kernel, you need to check out the correct source code,
cross-compile it, and then build the kernel image in the correct AOSP
directory.
Checking out kernel source code
Create a directory to store the kernel source code and clone the AOSP kernel git
repository to your local storage.
mkdir ~/src/marlin-kernel-src
cd ~/src/marlin-kernel-src
git clone https://android.googlesource.com/kernel/msm
After you are done, there should be an empty directory named msm.
Enter the msm directory and git checkout the branch
that corresponds to the source code you are building. For the list of available
branches and tags, see the Android msm
kernel source tree.
cd msm
git checkout TAG_NAME
After completing this step, the msm directory should have content.
Performing cross compilation
Next you need to compile the Android kernel.
Setting up your cross-compiler
To build the kernel, you need to set up a cross-compiler. The current
recommended and tested toolchain is Android's NDK toolchain latest stable
version. To download the Android NDK, visit the official Android NDK
website. Download the appropriate zip archive for your platform, and unzip
it. This results in a directory resembling
android-ndk-NDK_VERSION.
Downloading the LZ4c tool
The Pixel kernel uses LZ4 compression,
so the lz4c tool is required when you build your kernel. If you
use Ubuntu, install the lz4c tool by:
sudo apt-get install liblz4-tool
Building your kernel
Set up your build environment from the marlin-kernel-src/msm
directory with:
export ARCH=arm64
export CROSS_COMPILE=PATH_TO_NDK/android-ndk-NDK_VERSION/toolchains/aarch64-linux-android-TOOLCHAIN_VERSION/prebuilt/linux-x86_64/bin/aarch64-linux-android-
Then build an unmodified version of your kernel to establish a working base:
make marlin_defconfig
make -j48
The result of the build process can be found at:
arch/arm64/boot/Image.lz4-dtb
Rebuilding the boot image in
AOSP
After you have built the kernel image, copy the result over into AOSP's
device/google/marlin-kernel directory, with:
cp ${marlin-kernel-src}/msm/arch/arm64/boot/Image.lz4-dtb device/google/marlin-kernel
source build/envsetup.sh
lunch aosp_marlin-userdebug
make -j48
After a successful build, flash the target device with:
cd out/target/product/marlin
fastboot flashall -w
After flashing, your device should boot. Verify the image you flashed to
the device is the kernel image you built by checking Kernel
version under Settings -> System -> About phone
after the device finishes booting.
Modifying the kernel
Enabling KASAN and KCOV compile
options
KASAN and KCOV codes are guarded by compilation flags, which are not turned on
for normal builds. To enable them, add KASAN and KCOV options to the config
file, but drop the LZ4 config.
To do this, make a copy of the default config file, for example,
marlin_defconfig:
cd arch/arm64/configs
cp marlin_defconfig marlin-kasan_defconfig
In the new config file, remove this flag CONFIG_KERNEL_LZ4=y and
add these flags:
CONFIG_KASAN=y
CONFIG_KASAN_INLINE=y
CONFIG_KCOV=y
CONFIG_SLUB=y
CONFIG_SLUB_DEBUG=y
Recompiling the kernel
with new configuration
After you've finished modifying your copy of the config file, recompile the
kernel.
Reconfiguring the kernel
Set up your build environment.
Build your modified defconfig and check if the newly added flags
are present in the produced .config file.
make marlin-kasan_defconfig
grep KASAN .config
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_KASAN=y
# CONFIG_KASAN_OUTLINE is not set
CONFIG_KASAN_INLINE=y
You should see the KASAN flags. Compile your kernel:
make -j48
Checking the modified kernel
image
After a successful compilation, navigate to the arch/arm64/boot
directory to view the compilation results. Generally, the
Image.gz-dtb is about 23MB and larger than that of a standard build.
cd arch/arm64/boot
ls -lh Image.gz-dtb
-rw-r--r-- 1 username groupname 23M Aug 11 13:59 Image.gz-dtb
To see if KCOV was properly compiled, perform additional analysis on the
produced vmlinux at the root of the kernel source tree. If you run
an objdump on vmlinux, you should see numerous calls to
__sanitizer_cov_trace_pc().
sh -c '${CROSS_COMPILE}objdump -d vmlinux' | grep sanitizer
ffffffc000082030: 94040658 bl ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082050: 94040650 bl ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082078: 94040646 bl ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc000082080: 94040644 bl ffffffc000183990 <__sanitizer_cov_trace_pc>
ffffffc0000820ac: 94040639 bl ffffffc000183990 <__sanitizer_cov_trace_pc>
Modifying AOSP code
Before plugging in the new boot image, you need to adjust certain parameters in
AOSP's source code that govern how the device boots. This is mainly necessary to
ensure the new (inflated) image boots properly.
Adjusting board parameters
Adjust the boot parameters defined in the device's BoardConfig.mk
file. It is located at device/google/marlin/marlin relative to the
root of your AOSP source code.
cd device/google/marlin/marlin
vim BoardConfig.mk
Caution: Make sure you have a backup of the original
BoardConfig.mk file before proceeding, in case something goes
wrong.
The adjustments to be made can be summarized as follows through a
git diff result:
diff --git a/marlin/BoardConfig.mk b/marlin/BoardConfig.mk
index 31533fb9..81caf05d 100644
--- a/marlin/BoardConfig.mk
+++ b/marlin/BoardConfig.mk
@@ -116,15 +116,10 @@ BOARD_EGL_CFG := device/google/marlin/egl.cfg
BOARD_KERNEL_BASE := 0x80000000
BOARD_KERNEL_PAGESIZE := 4096
-ifneq ($(filter marlin_kasan, $(TARGET_PRODUCT)),)
BOARD_KERNEL_OFFSET := 0x80000
BOARD_KERNEL_TAGS_OFFSET := 0x02500000
BOARD_RAMDISK_OFFSET := 0x02700000
BOARD_MKBOOTIMG_ARGS := --kernel_offset $(BOARD_KERNEL_OFFSET) --ramdisk_offset $(BOARD_RAMDISK_OFFSET) --tags_offset $(BOARD_KERNEL_TAGS_OFFSET)
-else
-BOARD_KERNEL_TAGS_OFFSET := 0x02000000
-BOARD_RAMDISK_OFFSET := 0x02200000
-endif
TARGET_KERNEL_ARCH := arm64
TARGET_KERNEL_HEADER_ARCH := arm64
If you do not wish to modify the BoardConfig.mk
file, you could instead create a new lunch target that contains the name
marlin_kasan. For more information about this process, see
Adding a New Device.
Adjusting the kernel
target in the local Makefile
The new kernel uses LZ4 compression to improve speed, but KASAN requires gzip
for better compression ratio. To accommodate this, tell the build machinery
which kernel to bundle into the final target by modifying where the
LOCAL_KERNEL variable points to in
device/google/marlin/device-common.mk.
Rebuilding the boot image
To rebuild the boot image, copy the new kernel image into the AOSP tree in the
device-specific folder (e.g. device/google/marlin-kernel). Make
sure this is where the build system expects the kernel target image to be
at, according to how you modified it earlier.
Next, rebuild your flashable images, similar to how you built AOSP earlier. Upon successful build, flash all
built images as usual.
Booting your device
with a modified kernel image
You should now have a build that boots and enters the home screen. From here,
check the device's dmesg output for a "KernelAddressSanitizer
initialized" message in the very early boot stage. That message means
KASAN is initialized during boot time. Also, you can confirm
/sys/kernel/debug/kcov is present on the device (you will have to
be root to do that).
Troubleshooting
You can experiment with different kernel versions, starting with a standard
build as a working base, before turning on KASAN+KCOV compile options. When
things break, first check if the bootloader and baseband version on your device
matches those required by the new build. Finally, you might have to
catch up with a newer branch of the Android tree altogether if you venture too
far ahead with the kernel version.