docker中编译android aosp源码，出现Build sandboxing disabled due to nsjail error

使用docker搭建aosp的编译环境，测试中，出现Build sandboxing disabled due to nsjail error.解决办法如下。

1.错误现场

szhou@81fe32c25a6f:/home/builder/code/aosp$ source  build/envsetup.sh 
szhou@81fe32c25a6f:/home/builder/code/aosp$ lunch 

You're building on Linux

Lunch menu... pick a combo:
     1. aosp_arm-eng
     2. aosp_arm64-eng
     …… 省略 ……
     30. aosp_x86-eng
     31. aosp_x86_64-eng
     …… 省略 ……

Which would you like? [aosp_arm-eng] 31
16:23:33 Build sandboxing disabled due to nsjail error.
16:23:33 Build sandboxing disabled due to nsjail error.

============================================
PLATFORM_VERSION_CODENAME=S
PLATFORM_VERSION=S
TARGET_PRODUCT=aosp_x86_64
TARGET_BUILD_VARIANT=eng
TARGET_BUILD_TYPE=release
TARGET_ARCH=x86_64
TARGET_ARCH_VARIANT=x86_64
TARGET_2ND_ARCH=x86
TARGET_2ND_ARCH_VARIANT=x86_64
HOST_ARCH=x86_64
HOST_2ND_ARCH=x86
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.15.0-142-generic-x86_64-Ubuntu-14.04.5-LTS
HOST_CROSS_OS=windows
HOST_CROSS_ARCH=x86
HOST_CROSS_2ND_ARCH=x86_64
HOST_BUILD_TYPE=release
BUILD_ID=AOSP.MASTER
OUT_DIR=out
PRODUCT_SOONG_NAMESPACES=device/generic/goldfish device/generic/goldfish-opengl hardware/google/camera hardware/google/camera/devices/EmulatedCamera
============================================
szhou@81fe32c25a6f:/home/builder/code/aosp$

2. 错误分析

2.1 soong 错误打印

Z:\works\android\ustc\aosp\out\soong.log

2021/06/16 16:52:54.815584 build/soong/ui/build/build.go:184: Starting build with args: []
2021/06/16 16:52:54.815614 build/soong/ui/build/build.go:185: Environment: [HOSTNAME=81fe32c25a6f TRACE_BEGIN_SOONG=1623833572854843900 TERM=xterm ANDROID_PYTHONPATH=/home/builder/code/aosp/development/python-packages: OLDPWD=/home/builder/code/aosp OUT=/home/builder/code/aosp/out/target/product/generic_x86_64 TARGET_BUILD_VARIANT=eng BUILD_ENV_SEQUENCE_NUMBER=13 ANDROID_BUILD_PATHS=/home/builder/code/aosp/out/soong/host/linux-x86/bin:/home/builder/code/aosp/out/host/linux-x86/bin:/home/builder/code/aosp/prebuilts/gcc/linux-x86/x86/x86_64-linux-android-4.9/bin:/home/builder/code/aosp/development/scripts:/home/builder/code/aosp/prebuilts/devtools/tools:/home/builder/code/aosp/external/selinux/prebuilts/bin:/home/builder/code/aosp/prebuilts/misc/linux-x86/dtc:/home/builder/code/aosp/prebuilts/misc/linux-x86/libufdt:/home/builder/code/aosp/prebuilts/clang/host/linux-x86/llvm-binutils-stable:/home/builder/code/aosp/prebuilts/android-emulator/linux-x86_64:/home/builder/code/aosp/prebuilts/asuite/acloud/linux-x86:/home/builder/code/aosp/prebuilts/asuite/aidegen/linux-x86:/home/builder/code/aosp/prebuilts/asuite/atest/linux-x86: TOP=/home/builder/code/aosp TARGET_BUILD_APPS= TARGET_BUILD_TYPE=release PWD=/home/builder/code/aosp GCC_COLORS=error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01 HOME=/home/disk3/szhou SHLVL=1 ORIGINAL_PWD=/home/builder/code/aosp PYTHONPATH=/home/builder/code/aosp/development/python-packages: TARGET_GCC_VERSION=4.9 ANDROID_SOONG_HOST_OUT=/home/builder/code/aosp/out/soong/host/linux-x86 TARGET_PRODUCT=aosp_x86_64 OUT_DIR=out PYTHONDONTWRITEBYTECODE=1 TMPDIR=/home/builder/code/aosp/out/soong/.temp ASAN_SYMBOLIZER_PATH=/home/builder/code/aosp/prebuilts/clang/host/linux-x86/llvm-binutils-stable/llvm-symbolizer LANG=C.UTF-8 JAVA_HOME=/home/builder/code/aosp/prebuilts/jdk/jdk11/linux-x86 ANDROID_JAVA_HOME=prebuilts/jdk/jdk11/linux-x86 ANDROID_JAVA8_HOME=prebuilts/jdk/jdk8/linux-x86 ANDROID_JAVA9_HOME=prebuilts/jdk/jdk9/linux-x86 ANDROID_JAVA11_HOME=prebuilts/jdk/jdk11/linux-x86 PATH=/home/builder/code/aosp/prebuilts/jdk/jdk11/linux-x86/bin:/home/builder/code/aosp/prebuilts/jdk/jdk11/linux-x86/bin:/home/builder/code/aosp/out/soong/host/linux-x86/bin:/home/builder/code/aosp/out/host/linux-x86/bin:/home/builder/code/aosp/prebuilts/gcc/linux-x86/x86/x86_64-linux-android-4.9/bin:/home/builder/code/aosp/development/scripts:/home/builder/code/aosp/prebuilts/devtools/tools:/home/builder/code/aosp/external/selinux/prebuilts/bin:/home/builder/code/aosp/prebuilts/misc/linux-x86/dtc:/home/builder/code/aosp/prebuilts/misc/linux-x86/libufdt:/home/builder/code/aosp/prebuilts/clang/host/linux-x86/llvm-binutils-stable:/home/builder/code/aosp/prebuilts/android-emulator/linux-x86_64:/home/builder/code/aosp/prebuilts/asuite/acloud/linux-x86:/home/builder/code/aosp/prebuilts/asuite/aidegen/linux-x86:/home/builder/code/aosp/prebuilts/asuite/atest/linux-x86:/opt/mtk/neon_4.8.2_2.6.35_cortex-a9-ubuntu/x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin BUILD_DATETIME_FILE=out/build_date.txt]
2021/06/16 16:52:54.815674 build/soong/ui/build/build.go:161: Total RAM: 31.4GB
2021/06/16 16:52:55.013537 build/soong/ui/build/sandbox_linux.go:120: [prebuilts/build-tools/linux-x86/bin/nsjail -H android-build -e -u nobody -g nogroup -R / -B /home/builder/code/aosp -B /tmp -B /home/builder/code/aosp/out --disable_clone_newcgroup -- /bin/bash -c if [ $(hostname) == "android-build" ]; then echo "Android" "Success"; else echo Failure; fi]
2021/06/16 16:52:55.015716 build/soong/ui/build/sandbox_linux.go:127: Build sandboxing disabled due to nsjail error.
2021/06/16 16:52:55.015726 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mode: STANDALONE_ONCE
2021/06/16 16:52:55.015732 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Jail parameters: hostname:'android-build', chroot:'', process:'/bin/bash', bind:[::]:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0
2021/06/16 16:52:55.015735 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/' flags:MS_RDONLY type:'tmpfs' options:'' is_dir:true
2021/06/16 16:52:55.015737 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/' -> '/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' is_dir:true
2021/06/16 16:52:55.015740 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/home/builder/code/aosp' -> '/home/builder/code/aosp' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' is_dir:true
2021/06/16 16:52:55.015742 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/tmp' -> '/tmp' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' is_dir:true
2021/06/16 16:52:55.015744 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/home/builder/code/aosp/out' -> '/home/builder/code/aosp/out' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' is_dir:true
2021/06/16 16:52:55.015747 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Mount point: '/proc' flags:MS_RDONLY type:'proc' options:'' is_dir:true
2021/06/16 16:52:55.015749 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Uid map: inside_uid:65534 outside_uid:1007 count:1 newuidmap:false
2021/06/16 16:52:55.015751 build/soong/ui/build/sandbox_linux.go:130: [I][2021-06-16T16:52:55+0800] Gid map: inside_gid:65534 outside_gid:1007 count:1 newgidmap:false
2021/06/16 16:52:55.015755 build/soong/ui/build/sandbox_linux.go:130: [E][2021-06-16T16:52:55+0800][9596] bool subproc::runChild(nsjconf_t *, int, int, int)():447 clone(flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNET|SIGCHLD) failed. You probably need root privileges if your system doesn't support CLONE_NEWUSER. Alternatively, you might want to recompile your kernel with support for namespaces or check the current value of the kernel.unprivileged_userns_clone sysctl: Operation not permitted
2021/06/16 16:52:55.015758 build/soong/ui/build/sandbox_linux.go:130: [E][2021-06-16T16:52:55+0800][9596] int nsjail::standaloneMode(nsjconf_t *)():146 Couldn't launch the child process
2021/06/16 16:52:55.015761 build/soong/ui/build/sandbox_linux.go:136: nsjail failed with exit status 255
2021/06/16 16:52:55.015768 build/soong/ui/build/exec.go:64: "dumpvars" executing "prebuilts/build-tools/linux-x86/bin/ckati" [prebuilts/build-tools/linux-x86/bin/ckati -f build/make/core/config.mk --color_warnings --kati_stats dump-many-vars MAKECMDGOALS=]
2021/06/16 16:52:55.396789 build/soong/ui/build/exec.go:74: "dumpvars" finished with exit code 0 (381ms real, 346ms user, 47ms system, 363MB maxrss)
2021/06/16 16:52:55.396818 build/soong/ui/build/dumpvars.go:123: NINJA_GOALS droid
2021/06/16 16:52:55.396822 build/soong/ui/build/dumpvars.go:123: KATI_GOALS 

2.2 关键错误提示

从下面的错误看，已经提示权限不够，可能需要root权限。

• You probably need root privileges if your system doesn’t support CLONE_NEWUSER.
• sysctl: Operation not permitted

clone(flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNET|SIGCHLD) failed. You probably need root privileges if your system doesn't support CLONE_NEWUSER. Alternatively, you might want to recompile your kernel with support for namespaces or check the current value of the kernel.unprivileged_userns_clone sysctl: Operation not permitted

2.3 google结果

that does look docker-specific. Based on the nsjail readme, it looks like --privileged may be needed, which is unfortunate: https://github.com/google/nsjail#launching-in-docker

2.4 解决办法

添加 --privileged 参数

docker run --privileged  -it --user $(id -u ${USER}):$(id -g ${USER})   -v $PWD:/home/builder/code  -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /etc/shadow:/etc/shadow:ro   android_mm_build:latest /bin/bash

2.5 关于–privileged 参数

官网原文：https://docs.docker.com/engine/reference/run/

–privileged Give extended privileges to this container.

By default, Docker containers are “unprivileged” and cannot, for example, run a Docker daemon inside a Docker container. This is because by default a container is not allowed to access any devices, but a “privileged” container is given access to all devices (see the documentation on cgroups devices).

When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.

大致是说，不带此参数启动的容器，是不能访问host的device的，而使用了此参数之后，就如同有了root权限，可以像在host上访问device一样，访问各种设备、文件和配置。