【配置Omni-Swarm（omni swarm:开源的多机器人协同CSLAM算法）已完结】

配置Omni-Swarm（omni swarm:开源的多机器人协同SLAM算法）已完结

旨在记录配置Omni-Swarm过程，如需直接测试数据集，请跳转到可运行的Docker镜像

1 简介

Omni-swarm: A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swarms
Omni-swarm是一种用于空中群体的分布式全向视觉惯性超宽带（visual-inertial-UWB）状态估计系统。为了解决可观测性、复杂初始化、精度不足和缺乏全局一致性等问题，引入了全向感知系统作为Omni-swarm的前端，包括全向传感器，如立体鱼眼摄像头和超宽带（UWB）传感器，以及算法，包括鱼眼视觉惯性里程计（VIO）、多无人机基于地图的定位和视觉目标检测。该系统采用基于图优化和前向传递的方式作为Omni-swarm的后端，融合来自前端的测量数据。根据实验结果，所提出的分布式状态估计方法在群体系统上实现了厘米级的相对状态估计精度，并确保了全局一致性。此外，通过Omni-swarm的支持，无需任何外部设备就可以在完全分布式的方案中实现无人机之间的避障，展示了Omni-swarm在不同场景下成为自主空中群体飞行的基础的潜力。
论文地址
代码地址

---

2 配置过程

配置环境为Ubuntu18.04 amd64

2.1 ROS安装

最简便的ROS安装方法是采用鱼香ROS编写的fishROS一键安装脚本，使用过程如下
打开终端，运行以下代码：

sudo wget http://fishros.com/install -O fishros && sudo bash fishros

输入“1”以安装ROS，进到以下选择界面

如果自己换过源则选择“2”，未换过源的选择“1”自动换源（amd64为清华源）
按照指示换完源后到以下选择界面

选择“1”安装ROS1
图中为20.04示例，18.04对应的ros版本为melodic

选择“1”桌面版，等待安装完成

安装完成后运行roscore测试如下

ros安装完成

2.2 下载Omni-Swarm源码、CNN模型及数据集

1 源码下载
Omni-swarm可进入链接下载，也可使用git下载：

sudo apt install git
git clone https://github.com/HKUST-Aerial-Robotics/Omni-swarm.git

网络问题git不下来可以用以下命令git：

git clone https://mirror.ghproxy.com/https://github.com/HKUST-Aerial-Robotics/Omni-swarm.git

这种方式下载下来的是针孔相机版本，需要使用鱼眼相机版本于VINS-Fisheye网址下载源码
将git下来的包放入自己创建的工作空间，我的工作空间结构为：

/omni-swarm_ws/src/Omni-swarm-pinhole_compatiable

2 CNN模型下载
根据readme所述，须于此网址（科学上网）下载models文件夹
放于/Omni-swarm_ws/src/Omni-swarm/swarm_loop

3 数据集下载
此网址（科学上网）下载数据集
三个压缩包，共31.5g

2.3 下载依赖

根据README.md文件所述，需要下载安装：

• swarm_msgs
• inf_uwb_ros

参考此博客为了编译通过还需要安装以下未提到的依赖：

• libtorch
• TensorRT
• lcm
• libSGM
• yolo-tensorrt
• camera_models
• dw
• backward-cpp
• faiss
• graphviz

1 swarm_msgs和inf_uwb_ros
swarm_msgs和inf_uwb_ros直接git到主目录

git clone https://mirror.ghproxy.com/https://github.com/HKUST-Swarm/swarm_msgs.git
git clone https://mirror.ghproxy.com/https://github.com/HKUST-Swarm/inf_uwb_ros.git

2 libtorch和tensorRT
参考此博客
首先，查看cuda版本

nvcc --version

查看torch版本

python
>>import torch
>>print(torch.__version__)

根据以上版本在此博客选择对应的libtorch版本下载
在此博客选择对应的tensorRT版本下载，本机下载tensorRT7.1.3.4
将lib绝对路径添加到环境变量中：

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/d/Library/TensorRT-7.1.3.4.Ubuntu-18.04.x86_64-gnu.cuda-11.0.cudnn8.0/TensorRT-7.1.3.4/lib

为了避免其它软件找不到 TensorRT 的库，建议把 TensorRT 的库和头文件添加到系统路径下:

sudo cp -r ./lib/* /usr/lib
sudo cp -r ./include/* /usr/include

其使用需要修改omni-swarm的cmakelist，参考此博客

3 lcm
下载

git clone https://mirror.ghproxy.com/https://github.com/lcm-proj/lcmb.git

编译：

mkdir build
cd build/
cmake ..
make -j4
sudo make install    

4 yolo-tensorrt
下载dev-yolov4版本：

git clone https://mirror.ghproxy.com/https://github.com/enazoe/yolo-tensorrt/tree/dev-yolov4.git

修改cmakelist：

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14 -Wno-write-strings")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-rpath -Wl,$ORIGIN")
set(Torch_DIR "$ENV{HOME}/3rdParty/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED)
set(TENSORRT_ROOT $ENV{HOME}/3rdParty/TensorRT-7.1.3.4)
include_directories("$ENV{HOME}/3rdParty/TensorRT-7.1.3.4/include")

编译：

sudo apt-get install libgflags-dev
cd yolo-tensorrt/
mkdir build
cd build/
cmake ..
make

报错1：

原因:cuda版本错误，重装tensorRT相应版本的cuda（tensorRT7目前仅有cuda10.2版本以及cuda11.0版本）
重装后编译通过

5 dw

sudo apt-get install libdw-dev

6 backward-cpp

网址或git下载：

git clone https://mirror.ghproxy.com/https://github.com/bombela/backward-cpp.git

常规编译：

mkdir build
cd build/
cmake ..
make -j4
sudo make install 

7 faiss
安装参考此博客
1 安装依赖库openblas：

sudo apt-get install gfortran
git clone https://mirror.ghproxy.com/https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS
make FC=gfortran
sudo make install

sudo ln -s /opt/OpenBLAS/lib/libopenblas.so  /usr/lib/libopenblas.so
sudo cp libopenblas.so.0  /usr/lib/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/OpenBLAS/lib

2 安装依赖库lapack

# 下载lapack源码
wget http://www.netlib.org/lapack/lapack-3.4.2.tgz
tar -zxf lapack-3.4.2.tgz
# 首先当然是进入lapack-3.4.2文件夹，然后根据平台的特点，将INSTALL目录下对应的make.inc.XXX复制一份到 lapack-3.4.2目录下，并命名为make.inc, 这里我复制的是INSTALL/make.inc.gfortran，因为我这里用的是gfortran编译器
cd lapack-3.4.2
cp ./INSTALL/make.inc.gfortran ./
mv make.inc.gfortran make.inc

修改lapack-3.4.2/Makefile,因为lapack以来于blas库，所以需要做如下修改(注释第一句话，去掉注释第二句话):

#lib: lapacklib tmglib
lib: blaslib variants lapacklig tmglib

编译：
在lapack-3.4.2下运行：

# 编译所有的lapack文件
make
# 进入lapacke 文件夹，这个文件夹包含lapack的C语言接口文件 
cd lapacke
# 编译lapacke
make  

由于lapack的makefile文件中没有make isntall 命令，需要手工进行安装

# 将lapacke的头文件复制到系统头文件目录
sudo cp include/*.h /usr/include  
# 返回到 lapack-3.4.2 目录 
cd .. 
# 将生成的所有库文件复制到系统库目录 
sudo cp *.a /usr/lib 

3 编译安装faiss

#下载源码 // 这里注意代码路径
git clone https://mirror.ghproxy.com/https://github.com/RobotWithCV/faiss  
# 进入FAISS源码目录.
cd faiss
# 根据系统配置编译环境. [Linux 为例]
cp example_makefiles/makefile.inc.Linux ./makefile.inc

# 首先先执行./configure，看看环境是否符合编译条件，其中有一些不是必选项，一般只管输出日志的最后一句话有无错误即可
./configure

# 之后进行编译用例测试，若无报错即代表数学库安装成功
make misc/test_blas
./misc/test_blas

# 执行最重要的make & make install
make
sudo make install

8 graphviz

sudo apt install graphviz graphviz-dev

2.2 编译Omni-Swarm源码

前往Omni-swarm_ws运行：

catkin build

我首次编译只成功了4/9个包

一个一个找问题

问题1：

原因：缺少camera_models
解决方法：参考此博客下载VINS-Fisheye：

git clone https://mirror.ghproxy.com/https://github.com/HKUST-Aerial-Robotics/VINS-Fisheye.git

将里面的camera_models和vins_estimator复制到Omni的src中
同时下载bspline放入src中

git clone https://ghproxy.com/https://github.com/HKUST-Swarm/bspline

最终我的工作空间为这样：

问题2：
OpenCV版本小于3.4，我的OpenCV为3.3.1，现安装OpenCV，参考此博客
多版本OPENCV共存参考此博客

注意：需要安装OPENCV+Contrib+CUDA.编译过程会遇到很多问题，目前我CUDA版本11.0使用了很多方法都不成功，后续可能考虑降cuda版本

---

2024年1月16日更新
目前使用ubuntu20.04 Opencv4.8.0+cuda12.1 Swarm_loop包编译到最终的链接过程失败，opencv和ros链接不了，其他包编译通过。
报错如下：

/usr/bin/ld: CMakeFiles/swarm_loop_net_tester.dir/src/loop_network_tester.cpp.o: in function `void ros::Publisher::publish<swarmcomm_msgs::drone_network_status_<std::allocator<void> > >(swarmcomm_msgs::drone_network_status_<std::allocator<void> > const&) const':
/opt/ros/noetic/include/ros/publisher.h:107: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /opt/ros/noetic/include/ros/publisher.h:107: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /opt/ros/noetic/include/ros/publisher.h:107: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /opt/ros/noetic/include/ros/publisher.h:113: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /opt/ros/noetic/include/ros/publisher.h:113: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: CMakeFiles/swarm_loop_net_tester.dir/src/loop_network_tester.cpp.o:/opt/ros/noetic/include/ros/publisher.h:113: more undefined references to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)' follow
/usr/bin/ld: CMakeFiles/swarm_loop_net_tester.dir/src/loop_network_tester.cpp.o: in function `bool ros::NodeHandle::param<int>(std::string const&, int&, int const&) const':
/opt/ros/noetic/include/ros/node_handle.h:2122: undefined reference to `ros::NodeHandle::hasParam(std::string const&) const'
/usr/bin/ld: /opt/ros/noetic/include/ros/node_handle.h:2124: undefined reference to `ros::NodeHandle::getParam(std::string const&, int&) const'
/usr/bin/ld: CMakeFiles/swarm_loop_net_tester.dir/src/loop_network_tester.cpp.o: in function `main':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/loop_network_tester.cpp:67: undefined reference to `ros::init(int&, char**, std::string const&, unsigned int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/loop_network_tester.cpp:68: undefined reference to `ros::NodeHandle::NodeHandle(std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/loop_network_tester.cpp:70: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, std::string const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imshow(std::string const&, cv::_InputArray const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, double&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(sensor_msgs::Image_<std::allocator<void> > const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::error(int, std::string const&, char const*, char const*, int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/libloop_cnn.so: undefined reference to `loadTRTEngine(std::string, Logger&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imwrite(std::string const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, bool&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, std::string&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imencode(std::string const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::read(cv::FileNode const&, std::string&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::FileStorage::open(std::string const&, int, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `camodocal::CameraFactory::generateCameraFromYamlFile(std::string const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/swarm_loop_net_tester.dir/build.make:293：/home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/swarm_loop/swarm_loop_net_tester] 错误 1
make[1]: *** [CMakeFiles/Makefile2:1288：CMakeFiles/swarm_loop_net_tester.dir/all] 错误 2
make[1]: *** 正在等待未完成的任务....
/usr/bin/ld: CMakeFiles/swarm_loop_spy.dir/src/swarm_loop_spy.cpp.o: in function `SwarmLoopSpy::SwarmLoopSpy(ros::NodeHandle&)::{lambda(LoopEdge_t const&)#2}::operator()(LoopEdge_t const&) const':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:40: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: CMakeFiles/swarm_loop_spy.dir/src/swarm_loop_spy.cpp.o: in function `bool ros::NodeHandle::param<std::string>(std::string const&, std::string&, std::string const&) const':
/opt/ros/noetic/include/ros/node_handle.h:2122: undefined reference to `ros::NodeHandle::hasParam(std::string const&) const'
/usr/bin/ld: /opt/ros/noetic/include/ros/node_handle.h:2124: undefined reference to `ros::NodeHandle::getParam(std::string const&, std::string&) const'
/usr/bin/ld: /opt/ros/noetic/include/ros/node_handle.h:2122: undefined reference to `ros::NodeHandle::hasParam(std::string const&) const'
/usr/bin/ld: /opt/ros/noetic/include/ros/node_handle.h:2124: undefined reference to `ros::NodeHandle::getParam(std::string const&, std::string&) const'
/usr/bin/ld: CMakeFiles/swarm_loop_spy.dir/src/swarm_loop_spy.cpp.o: in function `main':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:83: undefined reference to `ros::init(int&, char**, std::string const&, unsigned int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:84: undefined reference to `ros::NodeHandle::NodeHandle(std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:80: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: CMakeFiles/swarm_loop_spy.dir/src/swarm_loop_spy.cpp.o: in function `SwarmLoopSpy::timer_callback(ros::TimerEvent const&)':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:67: undefined reference to `cv::putText(cv::_InputOutputArray const&, std::string const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:70: undefined reference to `cv::putText(cv::_InputOutputArray const&, std::string const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:72: undefined reference to `cv::imshow(std::string const&, cv::_InputArray const&)'
/usr/bin/ld: CMakeFiles/swarm_loop_spy.dir/src/swarm_loop_spy.cpp.o: in function `SwarmLoopSpy::SwarmLoopSpy(ros::NodeHandle&)::{lambda(FisheyeFrameDescriptor_t const&)#1}::operator()(FisheyeFrameDescriptor_t const&) const':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_spy.cpp:35: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, double&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(sensor_msgs::Image_<std::allocator<void> > const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::error(int, std::string const&, char const*, char const*, int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/libloop_cnn.so: undefined reference to `loadTRTEngine(std::string, Logger&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imwrite(std::string const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, bool&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imencode(std::string const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::read(cv::FileNode const&, std::string&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, int&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::FileStorage::open(std::string const&, int, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `camodocal::CameraFactory::generateCameraFromYamlFile(std::string const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/swarm_loop_spy.dir/build.make:286：/home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/swarm_loop/swarm_loop_spy] 错误 1
make[1]: *** [CMakeFiles/Makefile2:1545：CMakeFiles/swarm_loop_spy.dir/all] 错误 2
In file included from /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/swarm_loop.h:5,
                 from /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_nodelet.cpp:1:
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/loop_cam.h: In constructor ‘StereoFrame::StereoFrame(vins::FlattenImages, int)’:
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/loop_cam.h:64:27: warning: comparison of integer expressions of different signedness: ‘int’ and ‘std::vector<sensor_msgs::Image_<std::allocator<void> >, std::allocator<sensor_msgs::Image_<std::allocator<void> > > >::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
   64 |         for (int i = 1; i < vins_flatten.up_cams.size(); i++) {
      |                         ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/swarm_loop.h:5,
                 from /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_node.cpp:1:
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/loop_cam.h: In constructor ‘StereoFrame::StereoFrame(vins::FlattenImages, int)’:
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/include/swarm_loop/loop_cam.h:64:27: warning: comparison of integer expressions of different signedness: ‘int’ and ‘std::vector<sensor_msgs::Image_<std::allocator<void> >, std::allocator<sensor_msgs::Image_<std::allocator<void> > > >::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
   64 |         for (int i = 1; i < vins_flatten.up_cams.size(); i++) {
      |                         ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/bin/ld: CMakeFiles/swarm_loop_node.dir/src/swarm_loop_node.cpp.o: in function `main':
/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_node.cpp:16: undefined reference to `ros::init(int&, char**, std::string const&, unsigned int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_node.cpp:17: undefined reference to `ros::NodeHandle::NodeHandle(std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/Omni-swarm/swarm_loop/src/swarm_loop_node.cpp:18: undefined reference to `ros::console::set_logger_level(std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::putText(cv::_InputOutputArray const&, std::string const&, cv::Point_<int>, int, double, cv::Scalar_<double>, int, int, bool)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imshow(std::string const&, cv::_InputArray const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::console::initializeLogLocation(ros::console::LogLocation*, std::string const&, ros::console::levels::Level)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, double&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::hasParam(std::string const&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(sensor_msgs::Image_<std::allocator<void> > const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::error(int, std::string const&, char const*, char const*, int)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/libloop_cnn.so: undefined reference to `loadTRTEngine(std::string, Logger&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imwrite(std::string const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, bool&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv_bridge::toCvCopy(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, std::string&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::imencode(std::string const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::read(cv::FileNode const&, std::string&, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `ros::NodeHandle::getParam(std::string const&, int&) const'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `cv::FileStorage::open(std::string const&, int, std::string const&)'
/usr/bin/ld: /home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/liblibswarm_loop.so: undefined reference to `camodocal::CameraFactory::generateCameraFromYamlFile(std::string const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/swarm_loop_node.dir/build.make:293：/home/d/Workspace/Omni-swarm_ws/devel/.private/swarm_loop/lib/swarm_loop/swarm_loop_node] 错误 1
make[1]: *** [CMakeFiles/Makefile2:652：CMakeFiles/swarm_loop_node.dir/all] 错误 2
make: *** [Makefile:141：all] 错误 2

CMakeLists检查修改都无法解决。初步判断为C++17和C++14的链接冲突问题（我的libtorch必须使用C++17编译），还有待继续研究。

---

2024年1月17日更新
的确为libtorch导致的C++17和C++14的链接冲突问题
更换为docker中的libtorch版本（1.7.0+cpu，并未使用gpu，且为C++14）后编译通过
运行roslaunch

roslaunch swarm_loop nodelet-sfisheye.launch

存在问题：1.针孔相机roslaunch还有报错； 2.swarm_detector运行launch命令报错； 3.后续运行数据，跑出结果 4.docker没有运行成功，容器中没有找到src源码。

---

2024.3.15更新
swarm_detector包虽然编译通过，但运行launch命令报错

roslaunch swarm_detector detector.launch

错误信息：

runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.

报错原因是读取的engine模型为空，模型为/swarm_detector/config/drone_pose_v1_128x128_rtx3080_fp16.trt
分析原因是yolo-tensorrt中trt_utils.cpp中调用runtime->deserializeCudaEngine(）函数来解码模型

nvinfer1::IRuntime* runtime = nvinfer1::createInferRuntime(logger);
    nvinfer1::ICudaEngine* engine
        = runtime->deserializeCudaEngine(modelMem, modelSize/*, pluginFactory*/);

而模型由作者在trt7上创建，我用trt8解码导致报错。
解决方法：

使用tensorrt中的trtexec将config中的通用的onnx文件转换为trt8对应的engine，转换方法参考此博客
trtexec位于：

/Library/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/bin

运行命令为：

./trtexec --onnx=/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/swarm_detector/config/my_trans/drone_pose_v1.2_128x128.onnx --tacticSources=-cublasLt,+cublas --workspace=2048 --fp16 --saveEngine=/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/swarm_detector/config/my_trans/my_drone_pose.engine  --verbose

其中：
①–onnx 指定ONNX文件路径
②–tacticSources指定使用的方法库
③–workspace指定工作空间大小，单位是MB
④–fp16 开启FP16模式
⑤–saveEngine指定生成的engine的保存路径
⑥–verbose打开verbose模式，更多打印信息。
最后将生成的engine路径放入swarm_detector的launch文件中
swarm_detector的roslaunch通过。

---

2024.4.15更新

3 可运行的Docker镜像

可运行的Docker镜像
在原始的Omni-swarm docker（https://github.com/HKUST-Aerial-Robotics/Omni-swarm, TAG is xuhao1/swarm2020:pc）的基础上完成了项目的完整编译，形成了可直接运行的docker环境，测试结果：

docker镜像已push到dockerhub，TAG：blueberryfaygo/compiled_omniswarm:pc，通过以下命令拉取：

docker pull blueberryfaygo/compiled_omniswarm:pc

在https://github.com/HKUST-Aerial-Robotics/Omni-swarm基础上修改的数据集运行测试与评价工程已上传github与gitee，docker运行教程见其README。

4 报错汇总

编译Omni-swarm中三个包以及vins_estimator过程报错

1. ‘normalize’ is not a member of ‘cv::cuda’; did you mean ‘cv::normalize’
   解决方法：
   depth estimator.h添加

#include <opencv2/cudaarithm.hpp>
#include <opencv2/cudawarping.hpp>

2. ‘SparsePyrLKOpticalFlow’ is not a member of ‘cv::cuda’
   解决方法:
   feature_traker.h添加

#include <opencv2/cudaoptflow.hpp>

3. /home/d/Workspace/VINS-Fisheye/devel/lib/vins/vins_node_fisheye: error while loading shared libraries: libsgm.so: cannot open shared object file: No such file or directory
   问题原因：
   这个错误提示表明在运行vins_node_fisheye节点时，系统找不到名为libsgm.so的共享库文件。
   解决方法：
   如果你使用的是launch文件启动节点，在launch文件中添加一个env标签来设置LD_LIBRARY_PATH环境变量：

<env name="LD_LIBRARY_PATH" value="/path/to/libsgm.so:${LD_LIBRARY_PATH}" />

4. OPENCV4.2冲突
   解决方法：
   工程加入cv_bridge
5. CMake Error at /home/d/Library/libtorch2.1.0/share/cmake/Caffe2/public/cuda.cmake:65 (message):
   Found two conflicting CUDA installs:
   V12.1.66 in ‘/usr/local/cuda/include’ and
   V12.1.66 in ‘/usr/local/cuda-12.1/include’
   解决方法：打开libtorch中的cuda.cmake，注释63～68行
6. swarm_loop报错：error: #error You need C++17 to compile PyTorch
   解决：更换libtorch版本 改为C++14编译的版本（1.7.0+cpu）
7. error: call of overloaded ‘asin(double)’ is ambiguous
   解决：添加std::asin;根据报错信息，有时是添加ceres::sqrt
8. tensorrt_generic.cpp:16:43: error: invalid initialization of non-const reference of type ‘Logger&’ from an rvalue of type ‘std::nullptr_t’
   m_Engine = loadTRTEngine(engine_path, nullptr, m_Logger);
   解决：去掉nullptr
9. swarm_detector问题：
10. 找不到#include “munkres/munkres.h”
    github下载munkres-cpp 将src改名munkres后放入 include
11. YOLOV4-Tiny报错
    检查yolo-tensorrt中能用的yolo版本tag有没有YOLOV4-Tiny，没有需更换
12. /usr/bin/ld: 找不到 -lcamera_models
    /usr/bin/ld: 找不到 -lmunkres
    /usr/bin/ld: 找不到 -lloop_cnn
    解决：swarm_detector的CMakeLists，catkin中有以上三个包，直接把targetlink里的去掉

---

运行时错误

1. libtorch_docker/lib/libgomp-75eea7e8.so.1: version `GOMP_4.5’ not found (required by /usr/local/lib/libfaiss.so)
   问题原因：系统中的libgomp包含4.5版本，但工程直接去libtorch中找低版本libgomp
   解决方法：终端输入命令

ln -s /usr/lib/x86_64-linux-gnu/libgomp.so.1 /home/d/Library/libtorch_docker/lib/libgomp-75eea7e8.so.1

2. yolo-tensorrt make报错
   问题原因：tensorrt8不能用于低版本的yolo-tensorrt编译，须使用tensorrt7
   解决方法：使用docker中编译好的yolo-tensorrt
3. Swarm-Detector一系列错误
   1 runtime：
   ERROR: 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
   解决方法：yolo-tensorrt/modules/trt_utils.cpp 第353、354行注释

// free(modelMem);
// runtime->destroy();

2 nodelet: /home/d/Library/yolo-tensorrt/modules/yolo.cpp:1310: bool Yolo::verifyYoloEngine(): Assertion `get3DTensorVolume(m_Engine->getBindingDimensions(tensor.bindingIndex)) == tensor.volume && “Tensor volumes dont match between cfg and engine file \n”’ failed.
解决方法：yolo-tensorrt/modules/yolo.cpp第1161行～1166行注释

// if (m_NetworkType == "yolov4")//pan
// {
//  outputTensor.gridSize = (m_InputH / 32) * pow(2, 2-_n_yolo_ind);
//  outputTensor.grid_h = (m_InputH / 32) * pow(2, 2-_n_yolo_ind);
//  outputTensor.grid_w = (m_InputW / 32) * pow(2, 2-_n_yolo_ind);
// }

3 运行launch报错

roslaunch swarm_detector detector.launch

导入.trt文件时显示为空，溯源为yolo-tensorrt中trt_utils.cpp: loadTRTEngine（）函数，调用了Tensorrt中NvInferRuntime.h中nvinfer1：ICudaEngine* deserializeCudaEngine（）函数

其中问题1：参数IPluginFactory* pluginFactory无法读取，由于是可选参数，注释掉。
问题2：yolo-tensorrt读不到tensorrt的IPluginFactory类
解决：将yolo-tensorrt中整个类注释掉

问题3：
Yolo::verifyYoloEngine(): Assertion `volSize == tensor.volume && “Tensor volumes dont match between cfg and engine file \n”’ failed.
原因：yolo版本不匹配，选择的版本为yolov4，而导入的模型为yolov4-tiny
解决：swarm_detector/include/swarm_detector/tensorrt_detector.hpp 第27、28行，config.net_type选择yolov4_tiny

问题4：loading TRT pose engine时报错：
runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
原因：trt engine是tensorrt7建造的 但现在解码使用的tensorrt8

解决：使用tensorrt中的trtexec将config中的onnx文件转换为engine
trtexec位于

/Library/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0/TensorRT-8.6.1.6/targets/x86_64-linux-gnu/bin

运行命令为：

./trtexec --onnx=/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/swarm_detector/config/my_trans/drone_pose_v1.2_128x128.onnx --tacticSources=-cublasLt,+cublas --workspace=2048 --fp16 --saveEngine=/home/d/Workspace/Omni-swarm_ws/src/Omni-swarm-pinhole_compatiable/swarm_detector/config/my_trans/my_drone_pose.engine  --verbose

其中：
①–onnx 指定ONNX文件路径
②–tacticSources指定使用的方法库
③–workspace指定工作空间大小，单位是MB
④–fp16 开启FP16模式
⑤–saveEngine指定生成的engine的保存路径
⑥–verbose打开verbose模式，更多打印信息。
最后将生成的engine路径放入swarm_detector的launch文件中

---

Docker问题记录：

1. 找不到ros

source /opt/ros/melodic/setup.bash

2. 找不到camera_models中的config.cmake
   再编译一遍就行
3. vins项目报opencv错误
   修改vins_estimator中CMakelist，寻找对应版本opencv（默认为opencv3.4）,还不行就放入cv_bridge
4. cv::cuda找不到normalize等函数
   在对应文件中添加以下头文件：

#include <opencv2/cudaarithm.hpp>
#include <opencv2/cudawarping.hpp>

5. cv::cuda::SparsePyrLKOpticalFlow不是成员/没有声明
   在/featureTraker/feature_traker中添加头文件：

#include <opencv2/cudaoptflow.hpp>

6. fatal error: swarmtal_msgs/drone_onboard_command.h: No such file or directory(swarmcomm_msgs同理)
   在localization_proxy的CMakelists的find_package里添加swarmtal_msgs
7. /usr/bin/ld: libopencv_core.so版本冲突
   在工作空间配置一个cv_bridge
8. swarm_localization：libopencv_core.so.3.4: error adding symbols’ZN2cv5splitERKNS_3MatEPS0’: DSO missing from command line
   仿照swarm_loop把opencv和cv_bridge写入cmakelists
9. 找不到munkres/munkres.h
   github下载munkres-cpp 将src改名munkres后放入 include
10. /usr/bin/ld: cannot find -lcamera_models；/usr/bin/ld: cannot find -lmunkres；/usr/bin/ld: cannot find -lloop_cnn
    解决：swarm_detector的CMakeLists，catkin中有以上三个包，直接把targetlink里的去掉
11. Resource not found: swarm_pilot
    在swarm_localization的launch文件bag-replay.launch中，删掉：

<include file="$(find swarm_pilot)/launch/swarm_pilot.launch"> 
                <arg name="drone_id" default="$(arg drone_id)" />
                <arg name="output" default="log" />
            </include>
            <node name="gen_sim" type="gen_sim.py" pkg="swarm_pilot" />

12. m_engine == nullptr(swarm_loop)
    忘记下载readme中的CNNmodels到swarm_loop文件夹
13. /opt/ros/melodic/lib/nodelet/nodelet: symbol lookup error: /root/swarm_ws/devel/lib//libswarm_detector.so: undefined symbol: _ZN5ceres10CovarianceC1ERKNS0_7OptionsE
    问题原因：swarm_detector包中没有包含ceres
    解决方法：将findpackage、include_derection、target_link写入其CMakelist
14. Gtk-Message: 15:49:52.375: Failed to load module "canberra-gtk-module"ss
    在docker中：apt-get install libcanberra-gtk-module
15. libGL error: No matching fbConfigs or visuals found
    libGL error: failed to load driver: swrast
    解决方法：在config_all中的sh命令行添加以下：

export QMLSCENE_DEVICE=softwarecontext
export LIBGL_ALWAYS_INDIRECT=1

此贴完结。