Windows下Segment Any 3D Gaussians(SAGA)的配置及使用 --- 详细版
一、 项目准备
1、项目地址
https://github.com/Jumpat/SegAnyGAussians
2、下载压缩包
3、third_party文件下的模块
third_party文件下的模块需要单独下载,进入到文件夹中,下载zip压缩包解压
4、下载预训练模型和数据集
SAM预训练模型和数据集的下载在README.md中有链接,图中标蓝部分
总体的放置方式如下:
二、 项目配置
我的配置是:Cuda11.8+cub1.17.0+python3.10+torch2.2.0+pytorch3d0.7.6
1、pytorch环境
官网:PyTorch
由于版本与pytorch3d需要适配选择以前版本:Previous PyTorch Versions | PyTorch
例如cuda11.8+torch2.2.0命令如下:
pip install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu118
2、Pytorch3d环境(个人认为比较复杂)
①安装pytorch3d的依赖包
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install jupyter
pip install scikit-image matplotlib imageio plotly opencv-python
pip install black usort flake8 flake8-bugbear flake8-comprehensions
包括的包有:fvcore、iopath、cub、scikit-image、black、usort、flake8、matplotlib、tdqm、jupyter、imageio、plotly、opencv-python
②安装NVIDIA CUB
链接:NVIDIA CUB,下载对应安装包,解压后添加环境变量(用户变量和系统变量都添加了),我选择的1.17.0,对应版本如下:
③下载pytorch3d安装包
下载安装包:Pytorch3d
将安装包放置如下位置:
④安装VS2019
选择其他版本可能会报错,打开x64 Native Tools Command Prompt for VS 2019终端,conda activate xxx(自己的环境),进入到\pytorch3d-0.7.6,输入以下命令:
set DISTUTILS_USE_SDK=1
set PYTORCH3D_NO_NINJA=1
python setup.py install
检查环境中是否存在pytorch3d包
conda list pytorch3d
⑤遇到过的报错
File “D:\conda\envs\pytorch3d\lib\site-packages\torch\utils\cpp_extension.py”, line 1824, in_run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
原因:cub版本不正确(path路径和CUB_HOME是有区别的)
解决:修改cub的版本
CUB is now included in the CUDA Toolkit, so you no longer need to use your own checkout of CUB.
Define THRUST_IGNORE_CUB_VERSION_CHECK to ignore this. ball_query.cu
原因:cuda自带有cub不兼容需要屏蔽
解决:编辑位于 "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include\thrust\system\cuda\config.h"的config.h文件。找到 #ifndef THRUST_IGNORE_CUB_VERSION_CHECK 然后在它前面加上一行
#define THRUST_IGNORE_CUB_VERSION_CHECK
3、子模块
分别到目录下输入命令pip install -e.(如果重装了torch需要重新pip,且文件夹中多余的文件需要删除)
4、其他库
pip install plyfile==0.8.1
pip install numba
三、项目运行
1、特征提取
extract_features.py的第18行加上parser.add_argument(“–downsample”, default=“4”, type=str)
python extract_features.py --image_root <path to the scene data> --sam_checkpoint_path <path to the pre-trained SAM model> --downsample <1/2/4/8>
例如:python extract_features.py --image_root data/360_v2/kitchen --sam_checkpoint_path dependencies/sam_ckpt/sam_vit_h_4b8939.pth --downsample 4
2、生成mask
python extract_segment_everything_masks.py --image_root <path to the scene data> --sam_checkpoint_path <path to the pre-trained SAM model> --downsample <1/2/4/8>
例如:python extract_segment_everything_masks.py --image_root data/360_v2/kitchen --sam_checkpoint_path dependencies/sam_ckpt/sam_vit_h_4b8939.pth --downsample 4
3、预训练3DGS模型
python train_scene.py -s <path to COLMAP or NeRF Synthetic dataset>
例如:python train_scene.py -s data/360_v2/kitchen
会生成一个output/xxx预训练模型地址,其中xxx是随机生成的
4、训练3D特征
python train_contrastive_feature.py -m <path to the pre-trained 3DGS model>
例如:python train_contrastive_feature.py -m output/xxx
5、3D分割
方法1:在终端输入jupyter notebook进入网页,打开prompt_segmenting.ipynb,修改Hyper-parameters中的参数
DATA_ROOT = './data/360_v2/kitchen/'
MODEL_PATH = './output/XXX/'
FEATURE_GAUSSIAN_ITERATION = 30000
SAM_CKPT_PATH = './dependencies/sam_ckpt/sam_vit_h_4b8939.pth'
input_point = np.array([[200, 500]]) #提示点坐标,可以选多个
mask_id = 1 #初始有3个mask(0、1、2)
方法2:将prompt_segmenting.ipynb转换成prompt_segmenting.py
jupyter nbconvert --to script prompt_segmenting.ipynb
运行文件
python prompt_segmenting.py
6、渲染
python render.py -m <path to the pre-trained 3DGS model> --precomputed_mask <path to the segmentation results> --target scene --segment
例如:python render.py -m output/xxx --precomputed_mask segmentation_res/final_mask.pt --target scene --segment
7、查看
在3DGS下载viewer到your path\viewer
命令:.\viewer\bin\SIBR_gaussianViewer_app -m output/xxx
参考:
[1]pytorch3D Windows下安装经验总结_conda pytorch3d v windows-CSDN博客Windows下Pytorch3d的安装方法_pytorch3d安装-CSDN博客
[2]在windows环境下安装pytorch3d - 知乎 (zhihu.com)
[3]3D点云分割之SAGA(cvpr2023) 配置及使用_segment anything in 3d gaussians-CSDN博客
[4]SegAnyGAussians(SAGA)项目配置和运行-CSDN博客