Ubuntu20.04安装Kinect2驱动libfreenect2以及对应的ros功能包iai_kinect2(解决编译报错)

本人平台：nuc11-1165g7 Intel Xe Graphics

一.安装libfreenect2：

地址：https://github.com/OpenKinect/libfreenect2

1.安装libusb. The version must be >= 1.0.20：

sudo apt install -y libusb-1.0-0-dev

2.安装TurboJPEG：

sudo apt install -y libturbojpeg libjpeg-turbo8-dev libturbojpeg0-dev

3.安装OpenGL：

sudo apt install -y libglfw3-dev
sudo apt install -y libgl1-mesa-dev
sudo apt install -y libglu1-mesa-dev
sudo apt install -y freeglut3-dev
sudo apt install -y libglew1.8 libglew-dev
sudo apt install -y libgl1-mesa-glx
sudo apt install -y libxmu-dev

4.安装OpenNI2：

sudo apt install -y libopenni2-dev openni2-utils

5.安装OpenCL的头文件和库:

根据这篇文章(本人的流程有所不同)：Ubuntu 16.04.2 下为 Intel 显卡启用 OpenCL

1) 首先，安装官方的包：

sudo apt install ocl-icd-libopencl1
sudo apt install opencl-headers
sudo apt install clinfo
sudo apt install ocl-icd-opencl-dev
sudo apt install beignet-dev

其中clinfo是用来看系统opencl支持情况的，对于我11代Intel锐炬核显，Intel专用的opencl库就是beignet-dev(20.04,如果你是18.04这个包名是beignet)，如果你不安装这个包，终端输入clinfo回车是会告诉你没有匹配的设备的，只有安装了这个包，才会检测到opencl。
虽然libfreenect2仓库里的自述文件说opencl-headers对应AMD核心显卡，但是教程让我安我就安了，无所谓。

2) 接着下载官方的驱动：

https://software.intel.com/en-us/articles/opencl-drivers#latest_linux_SDK_release

！！从这之后跟教程不太一样！！

3) 进入Intel官方OpenCL仓库：

https://github.com/intel/compute-runtime/releases

下面这一套安装官方OpenCL的流程就在它仓库的自述文件里，随着库的更新命令可能会变化，建议伙伴们还是去它的仓库里按照它自己的流程走一遍命令

4) 创建一个暂时文件夹：

mkdir neo

5) 下载所有的deb包：

cd neo
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.13230.7/intel-igc-core_1.0.13230.7_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.13230.7/intel-igc-opencl_1.0.13230.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/intel-level-zero-gpu-dbgsym_1.3.25593.11_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/intel-level-zero-gpu_1.3.25593.11_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/intel-opencl-icd-dbgsym_23.05.25593.11_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/intel-opencl-icd_23.05.25593.11_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/libigdgmm12_22.3.0_amd64.deb

6)验证包的 sha256 总和：

wget https://github.com/intel/compute-runtime/releases/download/23.05.25593.11/ww05.sum
sha256sum -c ww05.sum

7) 安装所有包：

sudo dpkg -i *.deb

8) OpenCL安装结束（over）---------------------------------------------------------------------------------------

6.安装VAAPI （节选，只适用于intel，亲测安了之后，运行可执行文件就运行段错误，在此不给指令了）

7.安装一些其他的不知道啥用的包：

sudo apt install -y mesa-common-dev libxrandr-dev libxi-dev 

8.检查一下安装的环境：

1) 检查OpenCL：

clinfo

弹出：

Number of platforms                               1
  Platform Name                                   Intel(R) OpenCL HD Graphics
  Platform Vendor                                 Intel(R) Corporation
  Platform Version                                OpenCL 3.0 
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_byte_addressable_store cl_khr_device_uuid cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_command_queue_families cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_il_program cl_intel_mem_force_host_memory cl_khr_subgroup_extended_types cl_khr_subgroup_non_uniform_vote cl_khr_subgroup_ballot cl_khr_subgroup_non_uniform_arithmetic cl_khr_subgroup_shuffle cl_khr_subgroup_shuffle_relative cl_khr_subgroup_clustered_reduce cl_intel_device_attribute_query cl_khr_suggested_local_work_size cl_intel_split_work_group_barrier cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_khr_3d_image_writes cl_intel_media_block_io cl_intel_va_api_media_sharing cl_intel_sharing_format_query cl_khr_pci_bus_info cl_intel_subgroup_local_block_io 
  Platform Host timer resolution                  1ns
  Platform Extensions function suffix             INTEL

  Platform Name                                   Intel(R) OpenCL HD Graphics
Number of devices                                 1
  Device Name                                     Intel(R) Iris(R) Xe Graphics
  Device Vendor                                   Intel(R) Corporation
  Device Vendor ID                                0x8086
  Device Version                                  OpenCL 3.0 NEO 
  Driver Version                                  23.05.25593.11
  Device OpenCL C Version                         OpenCL C 1.2 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Max compute units                               96
  Max clock frequency                             1300MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
    Supported affinity domains                    (n/a)
  Max work item dimensions                        3
  Max work item sizes                             512x512x512
  Max work group size                             512
  Preferred work group size multiple              64
  Max sub-groups per work group                   64
  Sub-group sizes (Intel)                         8, 16, 32
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 1 / 1       
    half                                                 8 / 8        (cl_khr_fp16)
    float                                                1 / 1       
    double                                               0 / 0        (n/a)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (n/a)
  Address bits                                    64, Little-Endian
  Global memory size                              13207846912 (12.3GiB)
  Error Correction support                        No
  Max memory allocation                           4294959104 (4GiB)
  Unified memory for Host and Device              Yes
  Shared Virtual Memory (SVM) capabilities        (core)
    Coarse-grained buffer sharing                 Yes
    Fine-grained buffer sharing                   No
    Fine-grained system sharing                   No
    Atomics                                       No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Preferred alignment for atomics                 
    SVM                                           64 bytes
    Global                                        64 bytes
    Local                                         64 bytes
  Max size for global variable                    65536 (64KiB)
  Preferred total size of global vars             4294959104 (4GiB)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        3932160 (3.75MiB)
  Global Memory cache line size                   64 bytes
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            268434944 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   4 bytes
    Pitch alignment for 2D image buffers          4 pixels
    Max 2D image size                             16384x16384 pixels
    Max planar YUV image size                     16384x16352 pixels
    Max 3D image size                             2048x2048x2048 pixels
    Max number of read image args                 128
    Max number of write image args                128
    Max number of read/write image args           128
  Max number of pipe args                         0
  Max active pipe reservations                    0
  Max pipe packet size                            0
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Max number of constant args                     8
  Max constant buffer size                        4294959104 (4GiB)
  Max size of kernel argument                     2048 (2KiB)
  Queue properties (on host)                      
    Out-of-order execution                        Yes
    Profiling                                     Yes
  Queue properties (on device)                    
    Out-of-order execution                        No
    Profiling                                     No
    Preferred size                                0
    Max size                                      0
  Max queues on device                            0
  Max events on device                            0
  Prefer user sync for interop                    Yes
  Profiling timer resolution                      52ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    Sub-group independent forward progress        No
    IL version                                    SPIR-V_1.2 
    SPIR versions                                 1.2 
  printf() buffer size                            4194304 (4MiB)
  Built-in kernels                                (n/a)
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_device_uuid cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_command_queue_families cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_il_program cl_intel_mem_force_host_memory cl_khr_subgroup_extended_types cl_khr_subgroup_non_uniform_vote cl_khr_subgroup_ballot cl_khr_subgroup_non_uniform_arithmetic cl_khr_subgroup_shuffle cl_khr_subgroup_shuffle_relative cl_khr_subgroup_clustered_reduce cl_intel_device_attribute_query cl_khr_suggested_local_work_size cl_intel_split_work_group_barrier cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_khr_3d_image_writes cl_intel_media_block_io cl_intel_va_api_media_sharing cl_intel_sharing_format_query cl_khr_pci_bus_info cl_intel_subgroup_local_block_io 

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Intel(R) OpenCL HD Graphics
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [INTEL]
  clCreateContext(NULL, ...) [default]            Success [INTEL]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Iris(R) Xe Graphics
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Iris(R) Xe Graphics
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Iris(R) Xe Graphics

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.11
  ICD loader Profile                              OpenCL 2.1
	NOTE:	your OpenCL library only supports OpenCL 2.1,
		but some installed platforms support OpenCL 3.0.
		Programs using 3.0 features may crash
		or behave unexpectedly

2) 检查OpenGL：

glxinfo | grep OpenGL

弹出：

OpenGL vendor string: Intel
OpenGL renderer string: Mesa Intel(R) Xe Graphics (TGL GT2)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 21.2.6
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.6 (Compatibility Profile) Mesa 21.2.6
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 21.2.6
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:

9.编译libfreenect2：

cmake里一些要修改的默认参数列出来了

cd libfreenect2

mkdir build && cd build

cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/freenect2 -DBUILD_PYTHON3=ON -DENABLE_CXX11=ON -DENABLE_VAAPI=OFF

make -j8

sudo make install

1) 安装结束

2) 拔下kinect2的电源或者USB

3) 修改libfreenect2/platform/linux/udev/90-kinect2.rules文件中的USB权限：

把所有0666，修改为0777，即最高权限。

4) 将kinect2的rules文件拷贝到udev目录下(build文件夹下open terminal)：

sudo cp ../platform/linux/udev/90-kinect2.rules /etc/udev/rules.d/

10.测试libfreenect2：

重新插上设备后，我按照libfreenect2仓库和iai_kinect2里的测试方法尝试：

# 在build路径下
./bin/Protonect
./bin/Protonect gl
./bin/Protonect cpu
./bin/Protonect cl

四个框都有图像就问题不大

二.安装iai_kinect2：

地址：https://github.com/code-iai/iai_kinect2.git
或者：https://gitcode.net/mirrors/code-iai/iai_kinect2

1.安装依赖（PCL、eigen3）：

sudo apt install libeigen3-dev libeigen3-doc pcl-tools

把iai_kinect2包放到已经添加到环境变量的ROS工作空间下，然后

2.检查依赖（用官方仓库的指令会弹错）：

rosdep install --from-paths /你自己的工作空间路径/src/iai_kinect2 --ignore-src -r

3.运行cmake：

这里freenect2_DIR参数最好自己指定
我这里指定了是因为我前面安装libfreenect2的时候指定了make install路径

catkin_make -DCMAKE_BUILD_TYPE="Release" -Dfreenect2_DIR=$HOME/freenect2/lib/cmake/freenect2 -DENABLE_CXX11=ON

4.编译：

catkin_make

不要担心，你肯定会报错

5.编译iai_kinect2最重要的事！！！！！！

贴一个我已经修改过的包（在我自己的硬件环境和软件环境下）:

链接: https://pan.baidu.com/s/1mb1GMAic2L5pK0j6SAp7Pg?pwd=5sqw
提取码: 5sqw

1) kinect2_bridge/CMakeLists.txt：

find_package(freenect2 REQUIRED HINTS "$ENV{HOME}/freenect2")

在其中我们能看见这个ros包默认make install出来的freenect2文件夹路径${HOME}/freenect2，对应前面的cmake -D参数

2) 本人编译过程中遇到的红色bug：

注意：需要修改这个包的源文件和cmake文件，最好用vscode打开iai_kinect2文件夹
注意：以下任何一个问题，修改了源文件以后，都要重新编译

(1) PCL库有关的各种error：

这个问题在kinect2_viewer文件夹下的CMakeLists.txt中，它寻找PCL包没有找到正确的位置，我在find_package(OpenMP)这行代码下增加其绝对路径

set(PCL_DIR /usr/include/pcl-1.10/pcl)
find_package(PCL REQUIRED)
include_directories(${PCL_INCLUDE_DIRS})

(2) CXX14 error：

同样是在kinect2_viewer文件夹下的CMakeLists.txt中，
在开头，把CHECK_CXX_COMPILER_FLAG("-std=c++11" COMPILER_SUPPORTS_CXX11)
改成CHECK_CXX_COMPILER_FLAG("-std=c++14" COMPILER_SUPPORTS_CXX14)
后面的一句也改为SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14")

(3) OpenCV各种宏定义找不到：

我直接一把梭，把所有include了opencv的cpp文件都添加了同样的库文件（用的装ros自动安装的opencv4.3）
添加

#include <opencv2/imgproc/types_c.h>
#include <opencv2/imgproc/imgproc_c.h>
#include <opencv2/imgcodecs/legacy/constants_c.h>
#include <opencv2/highgui/highgui.hpp>

到

kinect2_bridge/src/kinect2_bridge.cpp
kinect2_calibration/src/kinect2_calibration.cpp
kinect2_viewer/src/viewer.cpp

这些文件的include中

(4) 运行kinect2_bridge.launch报OpenCL registration is not available！的错误：

这个问题在iai_kinect2的issue里有人提到，是浮点数的问题
这里就是为什么要用vscode打开的原因，因为这个问题是在kinect2_registration/src/depth_registration.cl文件里被造成的
我使用vscode打开后，会提示你安装opencl有关的插件，安装完后，会给这个cl文件以高亮，
虽然issue中有人说只需要修改文件最后那几行中const float4 tmp = (float4)(sqrt(2.0));这段代码为const float4 tmp = (float4)(sqrt(2.0f));
但我还是把所有插件提示出的所有未加f后缀的浮点数都加上了f后缀，然后再重新catkin_make就好了。

但是，如果你不想用或者用不了OpenCL，你可以打开kinect2_bridge/launch/kinect2_bridge.launch文件
修改其中的一个参数<arg name="reg_method" default="default"/>为<arg name="reg_method" default="cpu"/>

也就是不使用OpenCL而是使用CPU。

三.测试iai_kinect2：

roslaunch kinect2_bridge kinect2_bridge.launch #连接传感器
rosrun kinect2_viewer kinect2_viewer kinect2 sd image #查看图像
rosrun kinect2_viewer kinect2_viewer kinect2 sd cloud #查看点云

如果都能正常打开，就说明没问题了。

四.用iai_kinect2标定：

遇见了，颜色和红外标定完，标定pose的时候输出值全为0的bug
这个bug在仓库的pr中被修改了：https://github.com/cbuchxrn/iai_kinect2_opencv4/commit/7123381392f2524f1f85713bb92ab1f2e768190f#diff-7f37e052948300fc492ebf0563b4630d3bdadfcf28020fb7d5e16c833994402c
可以搜这个标题：Added cam calibration and made changes to use with opencv4

暂时是用matlab那个工具箱标定了rgb相机，标定了好几次，重影问题还是很严重。

五.修改kinect2_bridge在rviz中显示的点云为垂直于xz平面的(仅在kinect2_link下生效)：

用vscode打开kinect2_bridge.cpp文件，查找setRPY，这个函数是设置kinect2_link的初始位姿的，将qZero.setRPY(0,0,0)改为qZero.setRPY(M_PI/2, M_PI, 0);，catkin_make然后运行kinect2_bridge.launch选中kinect2_link坐标系，就能看到点云已经转过来了，我也想修改其它坐标系下的点云位姿，但是稍微修改以下上面的那个rot变量，kinect2_link坐标系下的sd点云也会跟着变化，故放弃修改其它坐标系下的点云位姿了。

六.结束