Autonomous Emergency Landing for Multicopters using Deep Reinforcement Learning（2022IROS）

论文名称及关键词
名称：Autonomous Emergency Landing for Multicopters using Deep Reinforcement Learning（2022IROS）
关键词：reinforcement learning ，语义分割，深度补全

1. 系统框架
   由语义分割神经网络[ 18 ]生成环境的语义掩膜。深度补全网络[ 17 ]利用RGB图像和稀疏的深度信息输出周 围环境的稠密深度图。单目相机快照和稀疏深度种子足以外推语义掩码和深度图。2. 网络框架

2. 仿真训练概述
   

3. 奖励设计
   其中Rt为步长奖励，RS与视图中的语义类相关联，RC为机器人与环境的碰撞，RA为选择的动作（离散动作空间），RT为终端奖励。

4. 训练策略
   未解决的问题：飞行高度较高时，深度补全和立体匹配都不是绝对准确的。
   关于地形检测问题（陡坡）：
   从深度补全生成的深度图中提取的平坦度或倾斜度是有噪声的，使得这些方法中有效着陆点的识别变得困难。
   关于区域问题（水域陆地）：
   文章提到了可以用语义将形如(路面/道路、地形、水、树木、建筑物和汽车)等进行分类，但是实际场景设计在城市场景，未考虑该问题。

论文复现：
配置ubuntu18.04 python3.6
(ubuntu20.04直接按照作者给的readme文件就可顺利安装)
在ububtu18.04上会出现一些不兼容问题

总体环境配置按照作者给的流程：

1. Install the following dependencies:

   sudo apt update && sudo apt install build-essential cmake
   sudo add-apt-repository ppa:deadsnakes/ppa 
   sudo apt update && sudo apt install python3.6-dev python3.6-venv
   sudo apt install libeigen3-dev libyaml-cpp-dev libopencv-dev libpcl-dev liboctomap-dev libgoogle-glog-dev libglm-dev libvulkan-dev
   

2. Clone the repo:

   git clone git@github.com:VIS4ROB-lab/multirotors_landing.git
   

3. Navigate to the main library folder:

   cd multirotors_landing_lib
   

4. At this point, it is strongly recommended to create a virtual environment:

   python3.6 -m venv ~/venvs/landing/
   

5. Build the project with pip:

   source ~/venvs/landing/bin/activate
   pip install --upgrade pip wheel
   pip install opencv-python==4.5.2.54
   echo export ML_PATH=path_to_multirotors_landing >> ~/.bashrc  # Path without final "/"
   pip install .
   

   正常在ubuntu18.04环境下在 pip install . 这一步会报错
   问题分别是网络问题和库不兼容问题
   修改两个文件

   1. /home/lyh/multirotors_landing/multirotors_landing_lib/cmake/pybind11_download.cmake 文件下修改以下两行为在gitee下下载pybind11-v2.9.0
      GIT_REPOSITORY https://gitee.com/joosoo/pybind11.git
      GIT_TAG v2.9.0
   2. /home/lyh/multirotors_landing/multirotors_landing_lib/cmake/gtest_download.cmake 文件下修改
      GIT_REPOSITORY https://gitee.com/zqx5449/googletest.git
      修改后再进行
      pip install .

Building without Python Bindings

To build without Python bindings and use only the C++ library, run the following commands:

cd multirotors_landing_lib
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DPYTHON_EXECUTABLE=/usr/bin/python3.6 && make -j8

3D Models

The 3D models can be found here. Download them and untar the file in the folder multirotors_landing/meshes.

This can be done using the command line as follows:

cd multirotors_landing
mkdir meshes && cd meshes
filename="models_landing.tar.xz"
fileid="1xOTkeJnCde1AlD53VwStMAoEV7S4ZJa-"
html=`curl -c ./cookie -s -L "https://drive.google.com/uc?export=download&id=${fileid}"`
curl -Lb ./cookie "https://drive.google.com/uc?export=download&`echo ${html}|grep -Po '(confirm=[a-zA-Z0-9\-_]+)'`&id=${fileid}" -o ${filename}
tar -xvf models_landing.tar.xz
rm models_landing.tar.xz

Note: The models need to be stored in multirotors_landing/meshes, as this is the hard-coded expected location.

Every model is composed of three files:

1. *.obj: The 3D model geometry. This is used by the Vulkan-based renderer.
2. *_rgbs.png: The texture of the model. The first three channels of this image are the actual texture, while the alpha channel contains the semantic labels. This is used by the Vulkan-based renderer.
3. *.ply: Point cloud of the model used for collision checking during training. An OctoMap is generated from this file.

More information about the rendering pipeline can be found here.

Running Instructions

The configurations for training and testing are stored in multirotors_landing_lib/config.

Policy Training

To train the agent using ground-truth semantics and depth maps (use flag -h for more information):

cd multirotors_landing/multirotors_landing_rl
pip install -e .
python3 scripts/quadrotor_landing.py

Note the following:

• When training start, you should see 3 images from the drone’s perspective (RGB, depth, semantics) if the parameter common/visualization is set to true in the configuration file.
• To use more than one environment, adjust the number of environments and threads in the configuration file.
• The results of the training are stored in multirotors_landing/experiments, and the folder name is the timestamp at the start of the training.
• The training script saves the policy weights at constant intervals. The final model that should be used for testing is named last_step_model.zip.
• If the training is stopped by the user before it is complete, the latest policy weights are saved as interrupt_model.zip.

Evaluate Training Performance

To plot the results from training (use flag -h for more information):

python3 scripts/plot_training.py -f ${path_to_training_folder} --decay-lr

To find the best policy from a training run manually (notice that last_step_model.zip should be used for testing):

python3 scripts/find_best_policy_weights.py -f ${path_to_training_folder}

Policy Testing

To test a trained policy (use flag -h for more information):

python3 scripts/quadrotor_testing.py -w ${path_to_weights}

The results are stored in multirotors_landing/experiments/tests.

Pre-trained Policy

In the folder data we provide a pre-trained policy (4 environments, training and testing on an p3.2xlarge instance on AWS):

• In the folder data/trained_policy we store the logs and plots of the training, as well as the final policy weights.
• In the folder data/results_trained_policy we report the results of a test run the trained policy.

To use this trained policy, run:

python3 scripts/quadrotor_testing.py -w ../data/trained_policy/last_step_model.zip

Socket Communication

Note: In the current version of the code, communication sockets are not used, as we use the ground-truth images from the Vulkan-based renderer. However, we include the library we implemented for communication with semantic segmentation and depth completion neural networks for completeness. Unforuntaly, the networks we used in the paper are closed-source, so we cannot provide them.

Note: The provided examples can be used if the C++ code is built manually.

In this folder examples on how to use socket-based communication are provided. It is possible to run a series of examples, where different types of information are exchanged between a server and a client (e.g. strings, vectors, images). The provided sockets can be used to interface the Vulkan-based renderer with any custom neural network.

1. Communication between client and server via raw sockets:

    ./multirotors_landing_lib/build/server_socket # Terminal 1 -- server
    ./multirotors_landing_lib/build/client_socket # Terminal 2 -- client
   

2. Communication between client and server via communicators (which provide a more complete interface than raw sockets):

    ./multirotors_landing_lib/build/server_communicator # Terminal 1 -- server
    ./multirotors_landing_lib/build/client_communicator # Terminal 2 -- client
   

3. Python interface, with exchange of RGB and grayscale images:

    python3 multirotors_landing_lib/tests/communication/python/socket_server.py # Terminal 1 -- server
    ./multirotors_landing_lib/build/python_socket_client # Terminal 2 -- client
   

Contributing

Contributions that help to improve the code are welcome. In case you want to contribute, please adapt to the Google C++ coding style and run bash clang-format-all . on your code before any commit.

In particular, the nice-to-have list includes:

• Use more recent python version and libraries
• Interface the training and testing environments with open-source semantic nets