Linux：gcc交叉编译时设置了“--sysroot“会产生哪些影响

一、介绍

在做交叉编译的时候，常常涉及到一个gcc编译选项--sysroot,这个选项是用来设置目标平台根目录的。--sysroot选项的官方说明如下：

--sysroot=dir
Use dir as the logical root directory for headers and libraries. For example, if the compiler normally searches for headers in /usr/include and libraries in /usr/lib, it instead searches dir/usr/include and dir/usr/lib.

If you use both this option and the -isysroot option, then the --sysroot option applies to libraries, but the -isysroot option applies to header files.

The GNU linker (beginning with version 2.16) has the necessary support for this option. If your linker does not support this option, the header file aspect of --sysroot still works, but the library aspect does not.

从--sysroot的说明可以看出，其会对编译和链接过程中，查找头文件和链接库造成影响。

例如:
原本默认会从/usr/include目录中搜索头文件、从/usr/lib中搜索依赖库，
当设置了--sysroot=dir后则会从dir/usr/include搜索头文件、从dir/usr/lib中搜索依赖库。

通过gcc -print-search-dirs查看默认动态库搜索路径

$ ./aarch64-linux-gnu-gcc --sysroot=/home/admin/tx2-rootfs -print-search-dirs |
 grep libraries | sed 's/libraries: =//g' | tr ':' '\n' | xargs readlink -f
/usr/local/lib/linaro-7.3.1/lib/gcc/aarch64-linux-gnu/7.3.1
/usr/local/lib/linaro-7.3.1/lib/gcc/aarch64-linux-gnu
/usr/local/lib/linaro-7.3.1/lib/gcc
/usr/local/lib/linaro-7.3.1/aarch64-linux-gnu/lib/aarch64-linux-gnu
/usr/local/lib/linaro-7.3.1/aarch64-linux-gnu/lib64
/home/admin/tx2-rootfs/lib/aarch64-linux-gnu/7.3.1
/home/admin/tx2-rootfs/lib/aarch64-linux-gnu
/home/admin/tx2-rootfs/lib64
/home/admin/tx2-rootfs/usr/lib/aarch64-linux-gnu/7.3.1
/home/admin/tx2-rootfs/usr/lib/aarch64-linux-gnu
/home/admin/tx2-rootfs/usr/lib64
/usr/local/lib/linaro-7.3.1/aarch64-linux-gnu/lib
/home/admin/tx2-rootfs/lib
/home/admin/tx2-rootfs/usr/lib

gcc选项--sysroot对-I的影响

gcc官方文档关于-I依赖库搜索路径的介绍如下：

-I dir
-iquote dir
-isystem dir
-idirafter dir
Add the directory dir to the list of directories to be searched for header files during preprocessing. If dir begins with ‘=’ or $SYSROOT, then the ‘=’ or $SYSROOT is replaced by the sysroot prefix; see --sysroot and -isysroot.

Directories specified with -iquote apply only to the quote form of the directive, #include "file". Directories specified with -I, -isystem, or -idirafter apply to lookup for both the #include "file" and #include <file> directives.

You can specify any number or combination of these options on the command line to search for header files in several directories. The lookup order is as follows:

    1. For the quote form of the include directive, the directory of the current file is searched first.
    2. For the quote form of the include directive, the directories specified by -iquote options are searched in left-to-right order, as they appear on the command line.
    3. Directories specified with -I options are scanned in left-to-right order.
    4. Directories specified with -isystem options are scanned in left-to-right order.
    5. Standard system directories are scanned.
    6. Directories specified with -idirafter options are scanned in left-to-right order.
You can use -I to override a system header file, substituting your own version, since these directories are searched before the standard system header file directories. However, you should not use this option to add directories that contain vendor-supplied system header files; use -isystem for that.

The -isystem and -idirafter options also mark the directory as a system directory, so that it gets the same special treatment that is applied to the standard system directories.

If a standard system include directory, or a directory specified with -isystem, is also specified with -I, the -I option is ignored. The directory is still searched but as a system directory at its normal position in the system include chain. This is to ensure that GCC’s procedure to fix buggy system headers and the ordering for the #include_next directive are not inadvertently changed. If you really need to change the search order for system directories, use the -nostdinc and/or -isystem options.

经过测试和验证发现，

• -Idir编译器只会从dir路径下搜索头文件；
• -I=dir或-I$SYSROOT/dir则会受--sysroot影响。

gcc选项--sysroot对-L的影响

gcc官方文档关于-L依赖库搜索路径的介绍比较简单

-Ldir
Add directory dir to the list of directories to be searched for -l.

经过测试和验证，发现结果和-I选项类似

• -Ldir编译器只会从dir路径下搜索依赖库；
• -L=dir或-I$SYSROOT/dir则会受--sysroot影响。

使用CMAKE进行交叉编译时的建议

1. CMAKE设置--sysroot通过CMAKE_SYSROOT()配置

CMAKE官方文档推荐在工具链文件中设置，如用来交叉编译tx2的工具链文件tx2.cmake内容如下
 

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_VERSION 1)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(SYSROOT_PATH  /home/admin/tx2-rootfs)
set(CMAKE_SYSROOT "${SYSROOT_PATH}")
message(STATUS  "Using sysroot path as ${SYSROOT_PATH}")

set(CMAKE_STAGING_PREFIX /home/admin/workspace/staging/)
set(CMAKE_INSTALL_PREFIX /usr/local)

set(TOOLCHAIN_PATH /usr/local/lib/linaro-7.3.1)
set(TOOLCHAIN_HOST ${TOOLCHAIN_PATH}/bin/aarch64-linux-gnu)

set(TOOLCHAIN_CC "${TOOLCHAIN_HOST}-gcc")
set(TOOLCHAIN_CXX "${TOOLCHAIN_HOST}-g++")

set(CMAKE_C_COMPILER ${TOOLCHAIN_CC})
set(CMAKE_CXX_COMPILER ${TOOLCHAIN_CXX})

add_link_options("LINKER:-rpath-link,/home/admin/tx2-rootfs/lib/aarch64-linux-gnu:/home/admin/tx2-rootfs/usr/lib/aarch64-linux-gnu")

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

• 在执行cmake的时候指定该工具链文件，假如工具链文件的路径为/home/admin/workspace/tx2.cmake

$ cd /home/admin/workspace/myprj/build
$ cmake -DCMAKE_TOOLCHAIN_FILE=/home/admin/workspace/tx2.cmake ..

• 或者在你CMakeLists.txt定义project(myprj)前添加include(tx2.cmake)亦可

cmake_minimum_required (VERSION 3.19)
include(tx2.cmake)
project (myprj CXX)

2. CMAKE中用来添加头文件搜索路径的宏INCLUDE_DIRECTORIES()并不会受到--sysroot选项影响

这是因为如果想要--sysroot产生作用，则需要-I=include_dir或-I$SYSROOT/include_dir，而如果通过INCLUDE_DIRECTORIES(“$SYSROOT/include_dir”)则CMAKE将这个$SYSROOT/include_dir识别为一个相对于当前工程的相对路径，并且会在编译阶段将路径补全成相对于当前工程目录的绝对路径,例如当前工程路径为/home/admin/workspace/myprj在编译阶段传递到gcc的参数就变成了-I/home/admin/workspace/myprj/\$SYSROOT/include_dir而不是真正想要的-I$SYSROOT/include_dir

推荐做法
直接通过CMAKE本身的包管理工具find_package(pkg)然后将搜索到的包导出的头文件目录列表添加到include_directories(${pkg_INCLUDE_DIR})。
 

find_package(pkg)
include_directories(${pkg_INCLUDE_DIR})

注意：示例中查找的包名为pkg，则头文件路径会保存在名为pkg_INCLUDE_DIR的变量中,大小写敏感

3. 解决链接时无法找到动态库依赖嵌套问题

如果所编译的可执行程序所依赖的动态库也需要依赖其他动态库，但连接器没有发现，就会报下面错误。

warning: lib*.so.1, needed by lib*.so, not found (try using -rpath or -rpath-link)

解决此类问题的关键点

• 修复sysroot目录中/usr/lib/aarch64-linux-gnu中错误的动态库符号链接
• 添加链接选项-Wl,-rpath-link

原理分析

以找不到libm.so.6为例，到/home/admin/tx2-rootfs/usr/lib/aarch64-linux-gnu下查看libm.so
 

$ ls -la 
lrwxrwxrwx  1 admin admin       32 Jun  5 01:25 libm.so -> /lib/aarch64-linux-gnu/libm.so.6

发现libm.so指向了一个无效的链接，类似的无效符号链接有很多，为了保证以后链接不会出问题，需要：

• 修复这些符号链接到正确的路径
• 创建/lib/aarch64-linux-gnu符号链接

可是即使修复了指向无效动态库链接的问题，这个链接警告和报错还是没有实际解决。这时就要借助
-rpath和-rpath-link选项了
在CMAKE官方文档中，说可以通过设置CMAKE_INSTALL_RPATH和CMAKE_BUILD_WITH_INSTALL_RPATH这两个变量来向gcc增加-Wl,rpath,选项
 

set(CMAKE_INSTALL_RPATH "dir")
set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)

可是实际操作发现，只能设置-rpath而不能设置-rpath-link。
为了支持-rpath-link，推荐使用add_link_options增加-rpath-link选项

 add_link_options("LINKER:-rpath-link,/home/admin/tx2-rootfs/lib/aarch64-linux-gnu:/home/admin/tx2-rootfs/usr/lib/aarch64-linux-gnu")

add_link_option 宏在cmake 3.13版本以后才支持的
注意：由于--sysroot选项不会作用于-rpath-link，所以要填写完整路径名