Airsim+UnrealEngine4 搭建无人车/无人机自动驾驶仿真环境

1. Airsim 概述

AirSim是基于Unreal Engine引擎或者Unity引擎，打造的一个无人机/无人车的仿真平台，凭借着Unreal Engine的性能，能够出色的进行基于计算机视觉算法的autonomous vehicles的仿真，AirSim在使用时作为一个UE4的插件，可以嵌到任意Unreal environment中，所以我们可以根据需求实现不同任务，不同场景的仿真。

Our goal is to develop AirSim as a platform for AI research to experiment with deep learning, computer vision and reinforcement learning algorithms for autonomous vehicles. For this purpose, AirSim also exposes APIs to retrieve data and control vehicles in a platform independent way.
         ——Microsoft

微软想把它打造成人工智能研究的仿真平台，以无人驾驶为主(比如自主无人机、自动驾驶汽车)，在Airsim中跑深度学习，计算机视觉和强化学习算法。

2. Unreal Engine 概述

The Unreal Engine is a game engine developed by Epic Games.With its code written in C++, the Unreal Engine features a high degree of portability and is a tool used by many game developers today.The current release is Unreal Engine 4, designed for Microsoft Windows, macOS, Linux, SteamOS, HTML5, iOS, Android, Nintendo Switch, PlayStation 4, Xbox One and virtual reality (SteamVR/HTC Vive, Oculus Rift, PlayStation VR, Google Daydream, OSVR and Samsung Gear VR).

3. 在Linux下环境搭建

3.1. 手动 编译安装 UnrealEngine

UnrealEngine在github上私有的项目，https://github.com/EpicGames/UnrealEngine.git 不信你点点看。
但是只要你申请加入Unreal engine项目的组织，还是可以Clone下来的，具体步骤。

1、首先登陆UnrealEngine的缔造者Epic的官网，注册一个账号
2、然后在Epic的个人页面下–账号关联，关联到自己的GitHub账号 (这个关联表明你申请加入EpicGames的开发，他会向你发出一个加入邀请)
3、登陆GitHub，在https://github.com/EpicGames 确认邀请就可以了

接下来就按照Build It on Linux的步骤，下载、编译安装 就可以啦

git clone -b 4.18 https://github.com/EpicGames/UnrealEngine.git
cd UnrealEngine
//few times Epic folks broke the build so we will get commit that works
// git checkout af96417313a908b20621a443175ba91683c238c8
./Setup.sh
./GenerateProjectFiles.sh
 make
 //注意 UnrealEngine这个东西很占地方，不仅下载大，编译更大，所以你的HOME下至少要有20G的Free Space，不然等你编译的时候，空间占满了，就失败了，我开始就这样，后来把我的双系统Windows下的一个分区挂到home下的一个文件夹下，才解决。

3.2 编译airsim 插件

     # go to folder where you clone GitHub projects
     git clone https://github.com/Microsoft/AirSim.git
     cd AirSim
     ./setup.sh  // 配置依赖
     ./build.sh  //编译airsim库
     cd ros
     catkin_make // 编译airsim_ros接口
     

./Engine/Binaries/Linux/UE4Editor 执行这个这个路径下的可执行程序，启动Unreal Engine试试。有两点说明。

1. 如果在初始化启动的过程中出错了（比如OpenGL的错误），说明你要按照NVIDIA的显卡驱动
2. 这个软件很大，对显卡的显存要求也比较高

3.3 使用编译好的binaries环境+Airsim

These are experimental Linux binaries for Ubuntu 16.04
https://github.com/microsoft/AirSim/releases/tag/v1.2.0Linux

• Africa (safari-like environment with moving animals and poachers)
• Blocks
• City (large environment with moving vehicles and pedestrians)
• Forest
• LandscapeMountains
• Neighborhood (small urban neighbourhood block)
• SimpleMaze
• SubT
• TrapCam (hunting trap camera simulator with large and small game)
• Warehouse
• ZhangJiaJie

场景	描述
Africa.zip 1.05G
Blocks.zip 327 MB
CityEnviron.zip.001 1.66 GB
CityEnviron.zip.002 1.24 GB
Forest.zip 1.17 GB
LandscapeMountains.zip 657 MB
Neighborhood.zip 1.42 GB
SimpleMaze.zip 423 MB
TrapCam.zip.001 1.76 GB
TrapCam.zip.002 1.33 GB
Warehouse.zip 837 MB
ZhangJiaJie.zip 517 MB

Source code (zip)
Source code (tar.gz)

OK 至此，Airsim+UnrealEngine 在Linux下的环境就搭建好了，可以自己先体验一下。
后面再分享Airsim+UnrealEngine的使用

4. 如何启动 无人车/无人机自动驾驶仿真环境

要启动基于Airsim+UnrealEngine4 搭建无人车/无人机自动驾驶仿真环境，需要两个方面：

• airsim+UE4 也就是配置了airsim插件的UE4 environment
• airsim settings 不同的配置可以设置不同的仿真类型/传感器类型等
• airsim API， 有了以上两者，我们通过什么方式取回image, 得到vehicle的状态，并执行control cmd,这就要通过api与其进行数据交互。

首先关于airsim+UE4：

• 简单的方式，可以使用airsim发布的编译好的binaries环境，这种二进制文件，通过执行一个脚本可以直接运行。
• 另一种方式是自己编译UE4,以及enviromnent,并且在环境中添加airsim plugin，这种方式更加灵活，有时编译容易出错，说版本不匹配。

其次关于settings.json的设置：

AirSim Settings.json 很总要

最后是airsim API

airsim 采用RPC网络通信，所以可以通过创建rpc client与仿真器的rpc server进行交互。airsim提供了C++ /Python的接口用于基于api的程序开发。可以参考相应的C++ APIs Guide设计

[参考文档]

• APIs.md
• apis_cpp.md
• 深层次开发可以查看 code_structure.md

5. 最简单的方式配置示例

实现UE4+airsim 实现无人机在simple_flight mode下带有双目+深度的模式

- 场景, 使用linux下预先编译好的 Block.zip 
执行命令： ./Blocks.sh

- settign文件
- api
- 

6. 四种仿真类型

{
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor" // Car or ComputerVision
}

6.1 Hardware-in-Loop(HITL)

• flight controller runs in actual hardware such as Naze32 or Pixhawk chip.
• connect this hardware to PC using USB port
• Simulator talks to the device to retrieve actuator signals and send it simulated sensor data

{
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "Vehicles": {
    "PX4": {
      "VehicleType": "PX4Multirotor"
    }
  }
}

参考文档px4_setup.md

6.2 Software-In-Loop Simulation (SITL)

{
    "SettingsVersion": 1.2,
    "SimMode": "Multirotor",
    "Vehicles": {
        "PX4": {
            "VehicleType": "PX4Multirotor",
            "UseSerial": false
        }
    }
}

• firmware runs in your computer as opposed to separate board

参考文档 px4_sitl.md
Airsim SIL简单说明

6.3 airsim simple_flight （default）

参考simple_flight.md

• 特性：可以通过输入 angle rate, angle level, velocity or position的方式控制

6.4 Computer Vision (without flight controller)

{
  "SettingsVersion": 1.2,
  "SimMode": "ComputerVision"
}

• In this mode, physics engine is disabled and there is no vehicle, just cameras
• You can move around using keyboard (use F1 to see help on keys). Or you can call APIs to move cameras around and take images.
• You can press Record button to continuously generate images.

7. 获取传感器的类型和数据

参考文档 sensors.md

airsim 目前支持六种类型传感器

Camera 
Barometer = 1
Imu = 2
Gps = 3
Magnetometer = 4
Distance Sensor = 5
Lidar = 6

7.1 camera数据

• 首先指定哪个位置的摄像头，可以使用id或者name [“0”, “1”, “2”, “3”, “4”] or [front_center, front_right, front_left, bottom_center and back_center]
• 指定图像的类型/分辨率以及各种相机参数
  图像的类型指 [场景图/深度图/视差图/语义分割图等等吧]

// ImageType 0~7
  Scene = 0, 
  DepthPlanner = 1,   //(更类似于双目深度估计结果SGM) all points that are in plan parallel to camera have same depth
  DepthPerspective = 2,//(类似于depth_image_proc 来参生点云)using a projection ray that hits that pixel.
  DepthVis = 3, //用于深度可视化的 从近到远被量化成 从黑到白
  DisparityNormalized = 4,// 视差图，视差被归一化0-1  use (Xl - Xr)/Xmax
  Segmentation = 5, //场景的语义分割图
  SurfaceNormals = 6,
  Infrared = 7 //根据 object ID 映射成 grey scale 0-255

{
  "SettingsVersion": 1.2,
  "CameraDefaults": {
      "CaptureSettings": [
        {
          "ImageType": 0,
          "Width": 256,
          "Height": 144,
          "FOV_Degrees": 90,
          "AutoExposureSpeed": 100,
          "MotionBlurAmount": 0
        }
    ]
  },
  "SimMode": "ComputerVision"
}

参考image_apis.md
一个获取双目和深度图的示例

7.2 Lidar data

{
    "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings_json.md",
    "SettingsVersion": 1.2,

    "SimMode": "Multirotor",

     "Vehicles": {
		"Drone1": {
			"VehicleType": "simpleflight",
			"AutoCreate": true,
			"Sensors": {
			    "LidarSensor1": { 
					"SensorType": 6,
					"Enabled" : true,
					"NumberOfChannels": 16,
					"RotationsPerSecond": 10,
					"PointsPerSecond": 100000,
					"X": 0, "Y": 0, "Z": -1,
					"Roll": 0, "Pitch": 0, "Yaw" : 0,
					"VerticalFOVUpper": -15,
					"VerticalFOVLower": -25,
					"HorizontalFOVStart": -20,
					"HorizontalFOVEnd": 20,
					"DrawDebugPoints": true,
					"DataFrame": "SensorLocalFrame"
				},
				"LidarSensor2": { 
				   "SensorType": 6,
					"Enabled" : true,
					"NumberOfChannels": 4,
					"RotationsPerSecond": 10,
					"PointsPerSecond": 10000,
					"X": 0, "Y": 0, "Z": -1,
					"Roll": 0, "Pitch": 0, "Yaw" : 0,
					"VerticalFOVUpper": -15,
					"VerticalFOVLower": -25,
					"DrawDebugPoints": true,
					"DataFrame": "SensorLocalFrame"
				}
			}
		}
    }
}

参考文档 lidar.md

Aerial Informatics and Robotics Platform　官方Airsim说明文档

【1/编译安装】
Unreal Engine + AirSim/docs/build_linux.md
[Unreal Engine + AirSim/docs/build_windows.md]

【２/启动与使用】

camera_views.md
通过F1开关camera settings.json设置image mode 机器性能影响帧率

flight_controller.md
简要说明　Flight Controller　HIL SIL simple_flight PX4

hello_drone.md 介绍hello drone如何工作的
Airsim启动时，会启动RPC server，　Hello Drone uses the RPC client to connect it.

multi_vehicle.md　：Setting Up Multiple Vehicles

px4_build.md　　
make posix_sitl_default
./build_posix_sitl_default/src/firmware/posix/px4 ./posix-configs/SITL/init/ekf2/iris

px4_setup.md　PX4 Setup for AirSim
Setting up PX4 Hardware-in-Loop
Setting up PX4 Software-in-Loop

px4_sitl.md　详细讲Setting up PX4 Software-in-Loop

FAQ.md

log_viewer.md Log Viewer是一个数据调试工具

【３/深究Airsim】
design.md & paper

code_structure.md Airsim代码组织结构
AirLib包含（Physics engine、Sensor models、Vehicle models、Control library）
Unreal/Plugins/AirSim　Unreal Engine中用于自动驾驶的一个插件
MavLinkCom　　
Sample Programs　一些例程examples hellocar helloDrone Droneshell

custom_drone.md AirLib on a Real Drone

apis.md ——　c++ API　hello drone/ hello car
python.md——Python API接口使用
image_apis.md

reinforcement_learning.md Reinforcement Learning in AirSim

Remote Control.md 遥控的使用

Hardware-In-Loop Simulation)（HIL）

YouTube有关HIL using Airsim的视频教程
UE+AirSim& PX4+遥控器 在Windows环境下搭建

Pixhawk 有关硬件在环仿真的说明

---

强化学习demo
DeepQNeuralNetwork in AirSim using CNTK
https://github.com/Microsoft/AirSim/blob/master/PythonClient/DQNdrone.py

Autonomous vehicle navigation based on Deep Reinforcement Learnin using AirSim

参考文档：
Building On Linux