复现BundleSDF官方演示，一种对视频中的物体进行3D重建

文章地址:GitHub - NVlabs/BundleSDF: [CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects

他们提出了一种近乎实时的方法，用于从单目 RGBD 视频序列中对未知物体进行 6-DoF 跟踪，同时对该物体进行神经 3D 重建。我们的方法适用于任意刚性物体，即使在视觉纹理基本不存在的情况下也是如此。假设该物体仅在第一帧中被分割。不需要其他信息，也不对交互代理做出任何假设。我们方法的关键是神经对象场，它与姿势图优化过程同时学习，以便将信息稳健地积累成一致的 3D 表示，同时捕捉几何和外观。自动维护一个动态的姿势记忆帧池，以促进这些线程之间的通信。我们的方法可以处理具有大姿势变化、部分和完全遮挡、无纹理表面和镜面高光的具有挑战性的序列。我们在 HO3D、YCBInEOAT 和 BEHAVE 数据集上展示了结果，表明我们的方法明显优于现有方法。

复现开始

 第一步下载git仓库，下载zip到本地上解压

第二步下载数据

下载分割网络的预训练权重，并将其放在 

./BundleTrack/XMem/saves/XMem-s012.pth

 下载LoFTR outdoor_ds.ckpt 的预训练权重，并将其放在

./BundleTrack/LoFTR/weights/outdoor_ds.ckpt    

第三步准备环境

配置wsl2

搜索控制面板

点击程序

点击启用或关闭Windows功能

启用

Hyper-V

Windows 虚拟机监控程序平台

适用于Linux的 windows 子系统

虚拟机平台

配置好了重启。

在商店里搜索ubuntu

下载ubuntu 22.04

下载安装完毕，就能在搜索里启用

或者在PowerShell里切换

配置docker

下载docker desktop并安装

安装过程一直点确定就行。

使用docker需要注册账号，注册完账户，点击齿轮，打开Use the WSL 2 based engine

以下是我的配置环境

"experimental": false,
  "registry-mirrors": [
    "https://docker.mirrors.tuna.tsinghua.edu.cn",
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn"
  ]

打开此项以适应wsl2

Docker/环境设置

• 打开cmd，构建 docker 镜像（这只需执行一次，并且可能需要一些时间）。

cd BundleSDF-master
cd docker
docker build --network host -t nvcr.io/nvidian/bundlesdf .

如果报错了，估计是多次下载大量文件导致，可以打开dockerfile文件

除了第一行的其他的都注释上
然后重新运行

docker build --network host -t nvcr.io/nvidian/bundlesdf .

运行出来在docker 有此镜像

打开cmd运行

--name 后面是生成的容器名字，后面的nvcr.io是镜像的名字, /bin/bash是进入交互式命令

docker run -it --name my_container1 nvcr.io/nvidian/bundlesdf:latest /bin/bash

然后在交互式命令行里一行一行运行

注意最好下载安装好一个模块，保存为镜像，以免重新来过

一个模块安装完成后，exit退出交互式容器，再保存镜像

docker commit <container_id> my_image_1:tag

#设置时区
export TZ=US/Pacific
sudo ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ | sudo tee /etc/timezone

#安装依赖项
sudo apt-get update --fix-missing && \
sudo apt-get install -y libgtk2.0-dev wget bzip2 ca-certificates curl git vim tmux g++ gcc build-essential cmake checkinstall gfortran libjpeg8-dev libtiff5-dev pkg-config yasm libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine2-dev libv4l-dev qt5-default libgtk2.0-dev libtbb-dev libatlas-base-dev libfaac-dev libmp3lame-dev libtheora-dev libvorbis-dev libxvidcore-dev libopencore-amrnb-dev libopencore-amrwb-dev x264 v4l-utils libprotobuf-dev protobuf-compiler libgoogle-glog-dev libgflags-dev libgphoto2-dev libhdf5-dev doxygen libflann-dev libboost-all-dev proj-data libproj-dev libyaml-cpp-dev cmake-curses-gui libzmq3-dev freeglut3-dev

#下载并安装 Eigen
cd /
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz
tar xvzf./eigen-3.4.0.tar.gz
cd eigen-3.4.0
mkdir build
cd build
cmake..
sudo make install

#克隆 OpenCV 和 OpenCV Contrib
cd /
git clone https://github.com/opencv/opencv
cd opencv
git checkout 3.4.15
cd /
git clone https://github.com/opencv/opencv_contrib.git
cd opencv_contrib
git checkout 3.4.15

#构建 OpenCV（注意：cmake根据系统功能调整命令）

cd /opencv
mkdir build
cd build
cmake.. -DCMAKE_BUILD_TYPE=Release -DBUILD_CUDA_STUBS=OFF -DBUILD_DOCS=OFF -DWITH_MATLAB=OFF -Dopencv_dnn_BUILD_TORCH_IMPORTER=OFF -DCUDA_FAST_MATH=ON -DMKL_WITH_OPENMP=ON -DOPENCV_ENABLE_NONFREE=ON -DWITH_OPENMP=ON -DWITH_QT=ON -DWITH_OPENEXR=ON -DENABLE_PRECOMPILED_HEADERS=OFF -DBUILD_opencv_cudacodec=OFF -DINSTALL_PYTHON_EXAMPLES=OFF -DWITH_TIFF=OFF -DWITH_WEBP=OFF -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -DCMAKE_CXX_FLAGS=-std=c++11 -DENABLE_CXX11=OFF -DBUILD_opencv_xfeatures2d=ON -DOPENCV_DNN_OPENCL=OFF -DWITH_CUDA=ON -DWITH_OPENCL=OFF
make -j$(nproc)
sudo make install

#克隆PCL
cd /
git clone https://github.com/PointCloudLibrary/pcl
cd pcl
git checkout pcl-1.10.0
mkdir build
cd build
cmake.. -DCMAKE_BUILD_TYPE=Release -DBUILD_apps=OFF -DBUILD_GPU=OFF -DBUILD_CUDA=OFF -DBUILD_examples=OFF -DBUILD_global_tests=OFF -DBUILD_simulation=OFF -DCUDA_BUILD_EMULATION=OFF -DCMAKE_CXX_FLAGS=-std=c++11 -DPCL_ENABLE_SSE=ON -DPCL_SHARED_LIBS=ON
make -j$(nproc)
sudo make install

#安装其他库
sudo apt install -y libzmq3-dev freeglut3-dev

#克隆并构建 pybind11
cd /
git clone https://github.com/pybind/pybind11
cd pybind11
git checkout v2.10.0
mkdir build
cd build
cmake.. -DCMAKE_BUILD_TYPE=Release -DPYBIND11_INSTALL=ON -DPYBIND11_TEST=OFF
make -j$(nproc)
sudo make install

#克隆并构建 yaml-cpp
cd /
git clone https://github.com/jbeder/yaml-cpp
cd yaml-cpp
git checkout yaml-cpp-0.7.0
mkdir build
cd build
cmake.. -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Release -DINSTALL_GTEST=OFF -DYAML_CPP_BUILD_TESTS=OFF -DYAML_BUILD_SHARED_LIBS=ON
make -j$(nproc)
sudo make install

#安装 Miniconda
cd /
wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda.sh
/bin/bash /miniconda.sh -b -p /opt/conda
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc
source ~/.bashrc
/opt/conda/bin/conda update -n base -c defaults conda -y
/opt/conda/bin/conda config --set ssl_verify no
/opt/conda/bin/conda config --add channels conda-forge
/opt/conda/bin/conda create -n py38 python=3.8

#更新 PATH 并初始化 Conda
export PATH=$PATH:/opt/conda/envs/py38/bin
conda init bash
echo "conda activate py38" >> ~/.bashrc
conda activate py38

#安装 PyTorch 和其他 Python 包
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
pip install trimesh opencv-python wandb matplotlib imageio tqdm open3d ruamel.yaml sacred kornia pymongo pyrender jupyterlab ninja
conda install -y -c anaconda scipy

#克隆kaolin
cd /
git clone --recursive https://github.com/NVIDIAGameWorks/kaolin

#为 CUDA 设置环境变量
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
export OPENCV_IO_ENABLE_OPENEXR=1

#安装 FreeImage 二进制文件
imageio_download_bin freeimage

#建造kaolin
conda activate py38
cd /kaolin
FORCE_CUDA=1 python setup.py develop

#完成安装
pip install transformations einops scikit-image awscli-plugin-endpoint gputil xatlas pymeshlab rtree dearpygui pytinyrenderer PyQt5 cython-npm chardet openpyxl
sudo apt-get update --fix-missing
sudo apt install -y rsync lbzip2 pigz zip p7zip-full p7zip-rar

完成后为29G

然后打开ubuntu输入

最后一行要改成自己的镜像名字

DIR=$(pwd)/../
xhost + 
docker run --gpus all --env NVIDIA_DISABLE_REQUIRE=1 -it --network=host --name cs  --cap-add=SYS_PTRACE --security-opt seccomp=unconfined  -v /home:/home -v /tmp:/tmp -v /mnt:/mnt -v $DIR:$DIR  --ipc=host -e DISPLAY=${DISPLAY} -e GIT_INDEX_FILE 你的镜像名字 bash

成功进入交互式容器

运行bash build.sh

cd BundleSDF-master
bash build.sh

下载数据集

在此处找到示例牛奶数据

下载后的数据集放到debug文件夹下,home,bowen,debug为自己新建文件夹

然后原数据集太多数据，本台3060跑不完，我选择只留重要的关键帧

运行牛奶瓶数据集

• 运行您的 RGBD 视频（指定 video_dir 和所需的输出路径）。共有 3 个步骤。

  # 1) Run joint tracking and reconstruction
  python run_custom.py --mode run_video --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk --use_segmenter 1 --use_gui 1 --debug_level 2
  
  # 2) Run global refinement post-processing to refine the mesh
  python run_custom.py --mode global_refine --video_dir /home/bowen/debug/2022-11-18-15-10-24_milk --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk   # Change the path to your video_directory
  
  # 3) (Optional) If you want to draw the oriented bounding box to visualize the pose, similar to our demo
  python run_custom.py --mode draw_pose --out_folder /home/bowen/debug/bundlesdf_2022-11-18-15-10-24_milk

运行后结果有两个3D模型，一个大一个小

放到blender 看看

完成了

---

报错：

Could not find a package configuration file provided by "Qt5"

root@aa5894b3ca77:/opencv/build# cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_CUDA_STUBS=OFF -DBUILD_DOCS=OFF -DWITH_MATLAB=OFF -Dopencv_dnn_BUILD_TORCH_IMPORTE=OFF -DCUDA_FAST_MATH=ON  -DMKL_WITH_OPENMP=ON -DOPENCV_ENABLE_NONFREE=ON -DWITH_OPENMP=ON -DWITH_QT=ON -WITH_OPENEXR=ON -DENABLE_PRECOMPILED_HEADERS=OFF -DBUILD_opencv_cudacodec=OFF -DINSTALL_PYTHON_EXAMPLES=OFF  -DWITH_TIFF=OFF -DWITH_WEBP=OFF -DOPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -DCMAKE_CXX_FLAGS=-std=c++11 -DENABLE_CXX11=OFF  -DBUILD_opencv_xfeatures2d=ON -DOPENCV_DNN_OPENCL=OFF -DWITH_CUDA=ON -DWITH_OPENCL=OFF
-- Detected processor: x86_64
-- Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) (Required is at least version "2.7")
-- Looking for ccache - not found
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found suitable version "1.2.11", minimum required is "1.2.3")
Cleaning INTERNAL cached variable: JPEG_LIBRARY
Cleaning INTERNAL cached variable: JPEG_INCLUDE_DIR
-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR)
-- libjpeg-turbo: VERSION = 2.1.0, BUILD = opencv-3.4.15-libjpeg-turbo
-- Could NOT find Jasper (missing: JASPER_LIBRARIES) (found version "1.900.1")
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
CMake Error at cmake/OpenCVFindLibsGUI.cmake:18 (find_package):
  Could not find a package configuration file provided by "Qt5" with any of
  the following names:

    Qt5Config.cmake
    qt5-config.cmake

  Add the installation prefix of "Qt5" to CMAKE_PREFIX_PATH or set "Qt5_DIR"
  to a directory containing one of the above files.  If "Qt5" provides a
  separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  CMakeLists.txt:698 (include)


-- Configuring incomplete, errors occurred!
See also "/opencv/build/CMakeFiles/CMakeOutput.log".
See also "/opencv/build/CMakeFiles/CMakeError.log".

解决：

启动交互式容器后bash build.sh

bash build.sh