SplaTAM项目实践

一、工程目录介绍

二、Installation

2.1 conda分步式安装

conda create -n splatam python=3.10
conda activate splatam
conda install -c "nvidia/label/cuda-11.6.0" cuda-toolkit
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
pip install -r requirements.txt

这个过程中，如果服务器没有很好的配置conda，可能导致安装失败，特别是在pip install -r requirements.txt中。

针对pip install -r requirements.txt，可以使用如下傻瓜式安装：

pip install tqdm==4.65.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install Pillow -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install opencv-python -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install imageio -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install matplotlib -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install kornia -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install natsort -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install pyyaml -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install wandb -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install lpips -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install open3d==0.16.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install torchmetrics -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install cyclonedds -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install pytorch-msssim -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install open3d  -i https://pypi.tuna.tsinghua.edu.cn/simple

2.2 conda一键安装

使用提供的conda environment.yml文件安装：

conda env create -f environment.yml
conda activate splatam

注意：如果服务器没有很好的配置conda，也可能导致安装失败。

2.3 Docker(预提供)安装

docker pull nkeetha/splatam:v1
bash bash_scripts/docker_start.bash
cd /SplaTAM/
pip install virtualenv --user
mkdir venv
cd venv
virtualenv splatam --system-site-packages
pip install -r venv_requirements.txt

三、手机运行DEMO

（这一部分我没有实现，因为 NeRFCapture app无论如何都不能和服务器通信，就先不写）

四、下载通用数据集

默认的DATAROOT是 ./data 。

4.1 下载Replica

bash bash_scripts/download_replica.sh

这个数据集下载很快。

4.2 TUM-RGBD

bash bash_scripts/download_tum.sh

下载相对较慢。

4.3 ScanNet

这个数据集需要授权下载，自己慢慢摸索吧。

4.4 ScanNet++

这个数据集需要授权下载，自己慢慢摸索吧。

五、运行benchmark

强烈建议使用weights and biases  ，记录和监控项目运行log。

如果还没有wandb账户，清先创建一个：

   Weights & BiasesWeights & Biases, developer tools for machine learninghttps://wandb.ai/

如果使用wandb，请确保splatam.py中，大约473行的wandb.init中，设置entity的值是自己的wandb账户，如下所示：

在每个benchmark运行结束之后，自动计算the trajectory error和the rendering metrics。

评测结果默认保存在./experiments。

5.1 Replica

假设想在room0场景中运行SplaTAM，则运行：

python scripts/splatam.py configs/replica/splatam.py

假设想在room0场景中运行SplaTAM-s，则运行：

python scripts/splatam.py configs/replica/splatam_s.py

对于其他场景，仅仅修改configs/replica/splatam.py文件（配置文件）的指向即可。

其他数据集类似。

可能出现的问题：

问题1：ModuleNotFoundError: No module named 'diff_gaussian_rasterization'

(splatam) gvlib_ljh@gvlib-MS-ljh:~/VSLAM/SplaTAM$ bash bash_scripts/online_demo.bash configs/iphone/online_demo.py
net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
Traceback (most recent call last):
  File "/home/gvlib_ljh/VSLAM/SplaTAM/scripts/iphone_demo.py", line 30, in <module>
    from utils.eval_helpers import report_progress
  File "/home/gvlib_ljh/VSLAM/SplaTAM/utils/eval_helpers.py", line 10, in <module>
    from utils.recon_helpers import setup_camera
  File "/home/gvlib_ljh/VSLAM/SplaTAM/utils/recon_helpers.py", line 2, in <module>
    from diff_gaussian_rasterization import GaussianRasterizationSettings as Camera
ModuleNotFoundError: No module named 'diff_gaussian_rasterization'
Traceback (most recent call last):
  File "/home/gvlib_ljh/VSLAM/SplaTAM/viz_scripts/final_recon.py", line 16, in <module>
    from diff_gaussian_rasterization import GaussianRasterizer as Renderer
ModuleNotFoundError: No module named 'diff_gaussian_rasterization'
解决方法：

如果在pull SplaTAM 项目的时候加了--recursive参数之后，这个问题一般不会出现。

如：git clone https://github.com/spla-tam/SplaTAM.git  --recursive

否则就会出现这个问题（原因是SplaTAM 项目不完整）。

需要先安装3D Gaussian Splating项目。特别要安装3D Gaussian Splating项目中的diff_gaussian_rasterization库。

首先下载gaussian-splatting项目

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

一定要加上--recursive，否则会导致工程pull不完整。

然后在gaussian-splatting-main/gaussian-splatting-main/submodules/diff-gaussian-rasterization下执行：

python setup.py install

问题2：ModuleNotFoundError: No module named 'open3d'

(splatam) gvlib_ljh@gvlib-MS-ljh:~/VSLAM/SplaTAM$ bash bash_scripts/online_demo.bash configs/iphone/online_demo.py
net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
Traceback (most recent call last):
  File "/home/gvlib_ljh/VSLAM/SplaTAM/scripts/iphone_demo.py", line 30, in <module>
    from utils.eval_helpers import report_progress
  File "/home/gvlib_ljh/VSLAM/SplaTAM/utils/eval_helpers.py", line 10, in <module>
    from utils.recon_helpers import setup_camera
  File "/home/gvlib_ljh/VSLAM/SplaTAM/utils/recon_helpers.py", line 2, in <module>
    from diff_gaussian_rasterization import GaussianRasterizationSettings as Camera
ModuleNotFoundError: No module named 'diff_gaussian_rasterization'
Traceback (most recent call last):
  File "/home/gvlib_ljh/VSLAM/SplaTAM/viz_scripts/final_recon.py", line 14, in <module>
    import open3d as o3d
ModuleNotFoundError: No module named 'open3d'

解决方法：

pip3  install open3d  -i https://pypi.tuna.tsinghua.edu.cn/simple

问题3：TypeError: GaussianRasterizationSettings.__new__() missing 1 required positional argument: 'debug'

Traceback (most recent call last):
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 1007, in <module>
    rgbd_slam(experiment.config)
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 556, in rgbd_slam
    params, variables, intrinsics, first_frame_w2c, cam = initialize_first_timestep(dataset, num_frames,
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 187, in initialize_first_timestep
    cam = setup_camera(color.shape[2], color.shape[1], intrinsics.cpu().numpy(), w2c.detach().cpu().numpy())
  File "/opt/vslam/SplaTAM-main/utils/recon_helpers.py", line 14, in setup_camera
    cam = Camera(
TypeError: GaussianRasterizationSettings.__new__() missing 1 required positional argument: 'debug'
解决方法：

在/opt/vslam/SplaTAM-main/utils/recon_helpers.py的14行，原来是：

​

修改（在函数Camare中增加实参“debug=True”）为：

​

问题4：

wandb: Currently logged in as: huarzail. Use `wandb login --relogin` to force relogin
wandb: ERROR Error while calling W&B API: project not found (<Response [404]>)
Problem at: /opt/vslam/SplaTAM-main/scripts/splatam.py 472 rgbd_slam
wandb: ERROR It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no iss
如下：

​

解决方法：

方法（1）注释/opt/vslam/SplaTAM-main/scripts/splatam.py中472 左右的entity=config['wandb']['entity']

方法（2）修改/opt/vslam/SplaTAM-main/scripts/splatam.py中472 左右的entity=config['wandb']['entity']的值为自己的账户名称

​

问题5：
Traceback (most recent call last):                                                                                                      
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 1007, in <module>
    rgbd_slam(experiment.config)
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 840, in rgbd_slam
    loss, variables, losses = get_loss(params, iter_data, variables, iter_time_idx, config['mapping']['loss_weights'],
  File "/opt/vslam/SplaTAM-main/scripts/splatam.py", line 253, in get_loss
    im, radius, _, = Renderer(raster_settings=curr_data['cam'])(**rendervar)
ValueError: not enough values to unpack (expected 3, got 2)

​

解决方法：

修改splatam.py文件中的三个Render地方（原因可能是Render函数的返回值仅仅2个，但是这里却是3个，参数不一致）。

​

​

问题6：

在安装diff-gaussian-rasterization的时候，可能出现编译不通过，可以通过如下方法解决：

First, install gcc-10 & g++-10 globally (Conda binaries won’t be able to do the replacement),
then set environment variables to execute the compilation command like:

export CMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-10

CC=gcc-10 CXX=g++-10 python setup.py develop