[environ] 01. qemu搭建linux aarch64环境

介绍：通过qemu学习linux是非常有效的，本篇旨在搭建qemu linux aarch64环境，使用nfs访问rootfs。

host：ubuntu-22.04

qemu：7.0.0

linux：5.10

busybox：1.34.0

gcc-aarch64-linux-gnu：11.3.0

必坑：csdn code中的内容复制到命令行（例如，命令）或文本（例如，nfs配置）中可能不被识别（识别错误），肉眼看不出来，需要手动敲，要格外留意。

目录

工具链

编译qemu

安装

编译

编译linux

下载

编译

制作rootfs

下载

 编译

 制作rootfs

配置nfs 

启动

---

工具链

使用gcc-aarch64-linux-gnu作为交叉编译工具链，通过以下方法安装：

$ sudo apt install gcc-aarch64-linux-gnu 
$ aarch64-linux-gnu-gcc --version
aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

编译qemu

首先介绍apt安装qemu，再介从源码绍编译qemu，由于可能需要自定义设备，因此从源码编译qemu也是需要了解的。

安装

$ sudo apt install qemu
$ sudo apt install qemu-system-arm
$ qemu-system-aarch64 --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.3)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

编译

先从QEMU找到源码下载地址，下载及解压：

$ wget https://download.qemu.org/qemu-7.0.0.tar.xz
$ tar -xvf qemu-7.0.0.tar.xz

编译：

$ ./configure --target-list=aarch64-softmmu
$ make
$ ./build/qemu-system-aarch64 --version
QEMU emulator version 7.0.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

注：编译过程可能遇到没有依赖库的问题，如下：

ERROR: glib-2.48 gthread-2.0 is required to compile QEMU
$ sudo apt install libglib2.0-dev

ERROR: pixman >= 0.21.8 not present.
       Please install the pixman devel package.
$ sudo apt install libpixman-1-dev

编译linux

下载

源码可以从 The Linux Kernel Archives 找到，以下使用linux-5.10版本，下载并解压：

$ wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.10.tar.gz
$ tar -xvf linux-5.10.tar.gz
$ mv linux-5.10 linux

编译

参考以下方法编译，需要留意的是，使用defconfig不需要特别配置：

$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image -j3

制作rootfs

通过busybox制作rootfs。

下载

从BusyBox下载源码并解压：

$ wget https://busybox.net/downloads/busybox-1.34.0.tar.bz2
$ tar -xvf busybox-1.34.0.tar.bz2
$ mv busybox-1.34.0 busybox

 编译

编译的结果有动态链接和静态链接两种，先介绍动态链接方式，后介绍静态链接，笔者平时使用的环境是动态链接的：

$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- install

 制作rootfs

通过脚本制作rootfs，脚本命名为rootfs.sh，内容如下：

#!/bin/bash

busybox=${1}
rootfs=${2}

if [[ ! -d $busybox ]]; then
	echo "No busybox found!"
	echo "Usage: mkrootfs <busybox> <rootfs>"
	exit
fi

if [[ ! -d $rootfs ]]; then
	mkdir $rootfs
	echo "rootfs created!"
fi

cp $busybox/_install/*  $rootfs/ -rf
cd $rootfs

if [[ ! -d lib ]]; then
	mkdir lib
	cp /usr/aarch64-linux-gnu/lib/* lib/
fi

if [[ ! -d proc ]] && [[ ! -d sys ]] \
		&& [[ ! -d dev ]] && [[ ! -d mnt ]] \
		&& [[ ! -d var ]] && [[ ! -d etc/init.d ]]; then
	mkdir proc sys dev mnt var etc etc/init.d
fi

if [[ -f etc/init.d/rcS ]]; then
	rm etc/init.d/rcS
fi

echo "#!/bin/sh" > etc/init.d/rcS
echo "mount -t proc none /proc" >> etc/init.d/rcS
echo "mount -t sysfs none /sys" >> etc/init.d/rcS
echo "mount -t ramfs none /dev" >> etc/init.d/rcS
echo "/sbin/mdev -s" >> etc/init.d/rcS
chmod +x etc/init.d/rcS
echo "Finished!"

 执行脚本后将生成rootfs文件夹：

$ chmod a+x rootfs.sh 
$ ./rootfs.sh busybox rootfs
$ ls rootfs
bin  dev  etc  lib  linuxrc  mnt  proc  sbin  sys  usr  var

若使用静态链接，则需要menuconfig配置 CONFIG_STATIC=y，其余同动态链接。

 注：其实静态链接不需要rootfs.sh中的 cp /usr/aarch64-linux-gnu/lib/* lib/ ，但考虑到使用时有可能需要编译busybox之外的工具，所以最好也拷贝。

配置nfs 

以/home/andrew/Documents/virt/environ作为环境根目录，进行nfs配置：

$ sudo apt install nfs-kernel-server
$ sudo vim /etc/exports # add the following at the buttom
home/andrew/Documents/virt *(insecure,rw,sync,no_subtree_check)

$ sudo systemctl restart nfs-server
$ sudo exportfs
/home/andrew/Documents/virt
		<world>

$ showmount -e
Export list for ubuntu:
/home/andrew/Documents/virt *

启动

 创建启动脚本start.sh，其内容：

#! /bin/bash

qemu/build/qemu-system-aarch64 \
	-nographic \
	-M virt \
	-cpu cortex-a57 -smp 4 \
	-m 1024M \
	-kernel linux/arch/arm64/boot/Image \
	-device virtio-net-device,netdev=eth0 -netdev user,id=eth0 \
	-append "noinitrd root=/dev/nfs \
		 nfsroot=10.0.2.2:/home/andrew/Documents/virt/environ/rootfs,v3 rw \
		 init=/linuxrc ip=dhcp console=ttyAMA0"

 启动：

$ ./start.sh
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[    0.000000] Linux version 5.10.0 (andrew@ubuntu) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP PREEMPT Sun Nov 6 02:20:07 CST 2022
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 32 MiB at 0x000000007e000000
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x7ddf3b00-0x7ddf5fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.0
[    0.000000] percpu: Embedded 23 pages/cpu s56216 r8192 d29800 u94208
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: ARM erratum 832075
[    0.000000] CPU features: detected: ARM erratum 834220
[    0.000000] CPU features: detected: EL2 vector hardening
[    0.000000] CPU features: kernel page table isolation forced ON by KASLR
[    0.000000] CPU features: detected: Kernel page table isolation (KPTI)
[    0.000000] CPU features: detected: Spectre-v2
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line: noinitrd root=/dev/nfs 		 nfsroot=10.0.2.2:/home/andrew/Documents/virt/environ/rootfs,v3 rw 		 init=/linuxrc ip=dhcp console=ttyAMA0
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 963100K/1048576K available (14336K kernel code, 2796K rwdata, 7536K rodata, 5888K init, 518K bss, 52708K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv2m: range[mem 0x08020000-0x08020fff], SPI[80:143]
[    0.000000] random: get_random_bytes called from start_kernel+0x324/0x4e8 with crng_init=0
[    0.000000] arch_timer: cp15 timer(s) running at 62.50MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x1cd42e208c, max_idle_ns: 881590405314 ns
[    0.000289] sched_clock: 56 bits at 62MHz, resolution 16ns, wraps every 4398046511096ns
[    0.012562] Console: colour dummy device 80x25
[    0.016865] Calibrating delay loop (skipped), value calculated using timer frequency.. 125.00 BogoMIPS (lpj=250000)
[    0.017165] pid_max: default: 32768 minimum: 301
[    0.019293] LSM: Security Framework initializing
[    0.021801] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.021933] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.082327] /cpus/cpu-map: empty cluster
[    0.099306] rcu: Hierarchical SRCU implementation.
[    0.111434] EFI services will not be available.
[    0.115163] smp: Bringing up secondary CPUs ...
[    0.134715] Detected PIPT I-cache on CPU1
[    0.136236] CPU1: Booted secondary processor 0x0000000001 [0x411fd070]
[    0.157886] Detected PIPT I-cache on CPU2
[    0.158792] CPU2: Booted secondary processor 0x0000000002 [0x411fd070]
[    0.164759] Detected PIPT I-cache on CPU3
[    0.165090] CPU3: Booted secondary processor 0x0000000003 [0x411fd070]
[    0.166106] smp: Brought up 1 node, 4 CPUs
[    0.166225] SMP: Total of 4 processors activated.
[    0.166380] CPU features: detected: 32-bit EL0 Support
[    0.166500] CPU features: detected: CRC32 instructions
[    0.166715] CPU features: detected: 32-bit EL1 Support
[    0.597844] CPU: All CPU(s) started at EL1
[    0.602729] alternatives: patching kernel code
[    0.644669] devtmpfs: initialized
[    0.679164] KASLR enabled
[    0.691197] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.691722] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.699678] pinctrl core: initialized pinctrl subsystem
[    0.727934] DMI not present or invalid.
[    0.756772] NET: Registered protocol family 16
[    0.796225] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.796755] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.797731] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.798916] audit: initializing netlink subsys (disabled)
[    0.803856] audit: type=2000 audit(0.652:1): state=initialized audit_enabled=0 res=1
[    0.815722] thermal_sys: Registered thermal governor 'step_wise'
[    0.815828] thermal_sys: Registered thermal governor 'power_allocator'
[    0.818265] cpuidle: using governor menu
[    0.820468] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.822754] ASID allocator initialised with 32768 entries
[    0.833944] Serial: AMBA PL011 UART driver
[    0.940939] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 47, base_baud = 0) is a PL011 rev1
[    0.975033] printk: console [ttyAMA0] enabled
[    1.115200] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.116737] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    1.117445] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.117790] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    1.180069] cryptd: max_cpu_qlen set to 1000
[    1.291525] ACPI: Interpreter disabled.
[    1.302412] iommu: Default domain type: Translated 
[    1.332258] vgaarb: loaded
[    1.338315] SCSI subsystem initialized
[    1.348384] usbcore: registered new interface driver usbfs
[    1.350858] usbcore: registered new interface driver hub
[    1.356278] usbcore: registered new device driver usb
[    1.362496] pps_core: LinuxPPS API ver. 1 registered
[    1.363086] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.364424] PTP clock support registered
[    1.367503] EDAC MC: Ver: 3.0.0
[    1.406028] FPGA manager framework
[    1.408989] Advanced Linux Sound Architecture Driver Initialized.
[    1.461648] clocksource: Switched to clocksource arch_sys_counter
[    1.471674] VFS: Disk quotas dquot_6.6.0
[    1.472439] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.482486] pnp: PnP ACPI: disabled
[    1.652513] NET: Registered protocol family 2
[    1.677097] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes, linear)
[    1.678323] TCP established hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    1.696261] TCP bind hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    1.698359] TCP: Hash tables configured (established 8192 bind 8192)
[    1.703976] UDP hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.705201] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes, linear)
[    1.711673] NET: Registered protocol family 1
[    1.736011] RPC: Registered named UNIX socket transport module.
[    1.736849] RPC: Registered udp transport module.
[    1.737226] RPC: Registered tcp transport module.
[    1.737554] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.738119] PCI: CLS 0 bytes, default 64
[    1.765294] hw perfevents: enabled with armv8_pmuv3 PMU driver, 5 counters available
[    1.767184] kvm [1]: HYP mode not available
[    1.870682] Initialise system trusted keyrings
[    1.876098] workingset: timestamp_bits=42 max_order=18 bucket_order=0
[    1.922286] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.947501] NFS: Registering the id_resolver key type
[    1.948943] Key type id_resolver registered
[    1.949337] Key type id_legacy registered
[    1.951370] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.954496] 9p: Installing v9fs 9p2000 file system support
[    2.012822] Key type asymmetric registered
[    2.013457] Asymmetric key parser 'x509' registered
[    2.014236] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    2.014993] io scheduler mq-deadline registered
[    2.015589] io scheduler kyber registered
[    2.077832] pl061_gpio 9030000.pl061: PL061 GPIO chip registered
[    2.091154] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:
[    2.095069] pci-host-generic 4010000000.pcie:       IO 0x003eff0000..0x003effffff -> 0x0000000000
[    2.097351] pci-host-generic 4010000000.pcie:      MEM 0x0010000000..0x003efeffff -> 0x0010000000
[    2.098422] pci-host-generic 4010000000.pcie:      MEM 0x8000000000..0xffffffffff -> 0x8000000000
[    2.101796] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]
[    2.105792] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00
[    2.106798] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.107394] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    2.108022] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[    2.108424] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[    2.113627] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    2.143660] EINJ: ACPI disabled.
[    2.263957] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    2.280388] SuperH (H)SCI(F) driver initialized
[    2.290299] msm_serial: driver initialized
[    2.303112] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    2.402044] loop: module loaded
[    2.409433] megasas: 07.714.04.00-rc1
[    2.446254] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]
[    2.456308] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
[    2.467352] Intel/Sharp Extended Query Table at 0x0031
[    2.472570] Using buffer write method
[    2.475269] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]
[    2.479057] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
[    2.480376] Intel/Sharp Extended Query Table at 0x0031
[    2.492387] Using buffer write method
[    2.493050] Concatenating MTD devices:
[    2.501982] (0): "0.flash"
[    2.502353] (1): "0.flash"
[    2.502663] into device "0.flash"
[    2.599474] libphy: Fixed MDIO Bus: probed
[    2.605652] tun: Universal TUN/TAP device driver, 1.6
[    2.629541] thunder_xcv, ver 1.0
[    2.630360] thunder_bgx, ver 1.0
[    2.631069] nicpf, ver 1.0
[    2.640349] hclge is initializing
[    2.644099] hns3: Hisilicon Ethernet Network Driver for Hip08 Family - version
[    2.645540] hns3: Copyright (c) 2017 Huawei Corporation.
[    2.647231] e1000: Intel(R) PRO/1000 Network Driver
[    2.648051] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    2.649571] e1000e: Intel(R) PRO/1000 Network Driver
[    2.650329] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    2.651540] igb: Intel(R) Gigabit Ethernet Network Driver
[    2.652358] igb: Copyright (c) 2007-2014 Intel Corporation.
[    2.653445] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    2.654430] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    2.657258] sky2: driver version 1.30
[    2.664228] VFIO - User Level meta-driver version: 0.3
[    2.683369] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.683895] ehci-pci: EHCI PCI platform driver
[    2.684658] ehci-platform: EHCI generic platform driver
[    2.685423] ehci-orion: EHCI orion driver
[    2.693218] ehci-exynos: EHCI Exynos driver
[    2.709934] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.711081] ohci-pci: OHCI PCI platform driver
[    2.713198] ohci-platform: OHCI generic platform driver
[    2.714836] ohci-exynos: OHCI Exynos driver
[    2.719365] usbcore: registered new interface driver usb-storage
[    2.753537] rtc-pl031 9010000.pl031: registered as rtc0
[    2.756381] rtc-pl031 9010000.pl031: setting system clock to 2022-11-08T17:13:24 UTC (1667927604)
[    2.762228] i2c /dev entries driver
[    2.800361] sdhci: Secure Digital Host Controller Interface driver
[    2.801322] sdhci: Copyright(c) Pierre Ossman
[    2.804233] Synopsys Designware Multimedia Card Interface Driver
[    2.814496] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.840449] ledtrig-cpu: registered to indicate activity on CPUs
[    2.855311] usbcore: registered new interface driver usbhid
[    2.855988] usbhid: USB HID core driver
[    2.900610] NET: Registered protocol family 17
[    2.904921] 9pnet: Installing 9P2000 support
[    2.906430] Key type dns_resolver registered
[    2.910468] registered taskstats version 1
[    2.911187] Loading compiled-in X.509 certificates
[    2.940466] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    3.015424] Sending DHCP requests ., OK
[    3.054309] IP-Config: Got DHCP answer from 10.0.2.2, my address is 10.0.2.15
[    3.055001] IP-Config: Complete:
[    3.063751]      device=eth0, hwaddr=52:54:00:12:34:56, ipaddr=10.0.2.15, mask=255.255.255.0, gw=10.0.2.2
[    3.076262]      host=10.0.2.15, domain=, nis-domain=(none)
[    3.079889]      bootserver=10.0.2.2, rootserver=10.0.2.2, rootpath=
[    3.080012]      nameserver0=10.0.2.3
[    3.081291] 
[    3.112016] ALSA device list:
[    3.114102]   No soundcards found.
[    3.131186] uart-pl011 9000000.pl011: no DMA platform data
[    3.317191] VFS: Mounted root (nfs filesystem) on device 0:20.
[    3.324323] devtmpfs: mounted
[    3.427263] Freeing unused kernel memory: 5888K
[    3.438709] Run /linuxrc as init process
[    3.998122] random: fast init done

Please press Enter to activate this console. 
/ # poweroff
/ # umount: none busy - remounted read-only
umount: devtmpfs busy - remounted read-only
swapoff: can't open '/etc/fstab': No such file or directory
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[   15.209739] Flash device refused suspend due to active operation (state 20)
[   15.210529] Flash device refused suspend due to active operation (state 20)
[   15.212159] reboot: Power down