MMdetection3d环境搭建、使用MMdetection3d做3D目标检测训练自己的数据集、测试、可视化,以及常见的错误
1 mmdetection3d环境搭建与测试
1.1 从docker开始搭建环境
1.1.1 开始从docker环境搭建
1、克隆代码
git clone https://github.com.cnpmjs.org/open-mmlab/mmdetection3d.git
2、下载好代码后进入代码目录
cd mmdetection3d
3、开始使用docker构建mmdetection3d的镜像环境
docker build -t mmdetection3d docker/
构建mmdetection3d
镜像的过程中可能会报错
W: GPG error: https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu1804/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <cudatools@nvidia.com>
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release' is not signed.
The command '/bin/sh -c apt-get update && apt-get install -y ffmpeg libsm6 libxext6 git ninja-build libglib2.0-0 libsm6 libxrender-dev libxext6 && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100
- 1
- 2
- 3
在docker/Dockerfile文件FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel
的下面添加如下一行:RUN rm /etc/apt/sources.list.d/cuda.list
FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel
RUN rm /etc/apt/sources.list.d/cuda.list
- 1
- 2
4、成功构建mmdetection3镜像如下:
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ docker build -t mmdetection3d docker/
Sending build context to Docker daemon 3.072kB
Step 1/19 : ARG PYTORCH="1.6.0"
Step 2/19 : ARG CUDA="10.1"
Step 3/19 : ARG CUDNN="7"
Step 4/19 : FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel
---> bb833e4d631f
Step 5/19 : RUN rm /etc/apt/sources.list.d/cuda.list
---> Using cache
---> 984b1a43fc0d
Step 6/19 : ENV TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0+PTX"
---> Using cache
---> 3c68ebdb8a2f
Step 7/19 : ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
......
Stored in directory: /tmp/pip-ephem-wheel-cache-9v_li1t0/wheels/cc/fa/4a/067979eccddf6a22b46722493df8e07b0541956a5ab5bac8b1
Successfully built mmpycocotools
Installing collected packages: mmpycocotools
Attempting uninstall: mmpycocotools
Found existing installation: mmpycocotools 12.0.3
Uninstalling mmpycocotools-12.0.3:
Successfully uninstalled mmpycocotools-12.0.3
Successfully installed mmpycocotools-12.0.3
Removing intermediate container 3689dccd83f3
---> 2ee366dc3c2f
Successfully built 2ee366dc3c2f
Successfully tagged mmdetection3d:latest
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
使用docker images
可以查看到成功构建的镜像:
5、生成和启动mmdetection3容器
docker run --gpus all --shm-size=8g -it -v {DATA_DIR}:/mmdetection3d/data mmdetection3d
可以映射一个本地的路径到容器中,用于存放数据,这样不会导致你容器删除的时候出现数据丢失!!!
启动容器:
docker run --gpus all --shm-size=8g -it -v /home/shl/shl_res/MMlab/mmdetection3d/mmdetect3d_data:/mmdetection3d/data mmdetection3d
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ docker run --gpus all --shm-size=8g -it -v /home/shl/shl_res/MMlab/mmdetection3d/mmdetect3d_data:/mmdetection3d/data mmdetection3d
root@3672fb821035:/mmdetection3d# ls
LICENSE README.md build data docker mmdet3d requirements resources setup.py tools
MANIFEST.in README_zh-CN.md configs demo docs mmdet3d.egg-info requirements.txt setup.cfg tests
root@3672fb821035:/mmdetection3d#
- 1
- 2
- 3
- 4
- 5
1.1.2 测试demo
1、下载预训练模型
如果你要下载这个模型模型,直接点击下载
2、测试命令
python demo/pcd_demo.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py data/predtrain_models/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth --out-dir data/output_result/
3、测试结果:
root@3672fb821035:/mmdetection3d# tree data
data
|-- output_result
| `-- kitti_000008
| |-- kitti_000008_points.obj
| `-- kitti_000008_pred.obj
`-- predtrain_models
`-- hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth
3 directories, 3 files
root@3672fb821035:/mmdetection3d#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
在output_result目录下,会生成一个kitti_000008
的目录,目录下会生成目标的
4、对测试demo生成结果可视化
1.2 直接在外部安装mmdetection3d环境
1.2.1 创建并激活虚拟环境
conda create -n open-mmlab python=3.7 -y
conda activate open-mmlab
1.2.2 安装pytorch和torchvision
conda install pytorch torchvision -c pytorch
我安装的方式,是在[pypi]官网先下载好torch和torchvision
的whl安装包
,然后使用pip install xxx.whl
的方式安装!
注意:
在安装torch和torchvision的时候,二者的版本需要对应,版本对应关系如下:
1.2.3 安装mmcv
1、在安装mmcv之前,需要先确定你的:cuda
和torch
的版本,我的安装cuda和torch版本为:
cuda
:10.2
torch
:1.7.0
2、安装命令:
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html
然后把自己的cuda和torch的版本号替换即可,我的环境对应的具体命令为:
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.7.0/index.html
你也可以去上面的mmcv-full链接中,先把你需要的对应版本的mmcv-full的whl文件下载下来再安装!我安装的版本是:mmcv-full==1.3.1
1.2.4 安装MMDetection
安装MMDetection有两种方式:
- 直接使用pip安装
- 从MMDetection源码安装
1、直接使用pip安装
pip install git+https://github.com/open-mmlab/mmdetection.git
2、从MMDetection的源码安装(我是直接从源码安装的)
git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection
pip install -r requirements/build.txt
pip install -v -e . # or "python setup.py develop"
- 1
- 2
- 3
- 4
注意:
安装mmdet需要对应mmcv-full版本,具体如下表:
1.2.5 克隆编译mmdetection3d
1、克隆mmdetection3d仓库
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
- 1
- 2
2、编译mmdetection3d
pip install -v -e . # or "python setup.py develop"
注意:
编译的时候,我建议你使用命令:python setup.py develop
3、所有环境安装成功,导入相关库包,简单测试一下:
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ python
Python 3.7.10 (default, Feb 26 2021, 18:47:35)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> import mmcv
>>> import mmdet
>>> import mmdet3d
>>>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
1.3 测试demo
1、下载预训练模型
如果你要下载通过一个模型,直接点击下载
2、测试命令
python demo/pcd_demo.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py my_checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ python demo/pcd_demo.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py my_checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth
Use load_from_local loader
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ ls
- 1
- 2
- 3
3、测试结果:
默认生成结果会保存到./demo/
目录下,kitti_000008
就是生成的目录和结果,当然你也可以通过--out-dir参数
指定输出结果保存路径!
4、对测试demo生成结果可视化
2 数据转换和更高级的API
2.1 把ply格式文件转换为bin文件
1、如果你的点云数据是ply格式,可以通过下面的代码进行转换,但首先需要确保你已经安装了:pandas
和plyfile
库包:
import numpy as np
import pandas as pd
from plyfile import PlyData
def convert_ply(input_path, output_path):
plydata = PlyData.read(input_path) # read file
data = plydata.elements[0].data # read data
data_pd = pd.DataFrame(data) # convert to DataFrame
data_np = np.zeros(data_pd.shape, dtype=np.float) # initialize array to store data
property_names = data[0].dtype.names # read names of properties
for i, name in enumerate(
property_names): # read data by property
data_np[:, i] = data_pd[name]
data_np.astype(np.float32).tofile(output_path)
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
2、然后调用上面的转换函数即可:
convert_ply('./test.ply', './test.bin')
2.2 使用trimesh模块转换其他格式点云到ply格式
1、如果你的点云是其他格式,例如:off、obj等,可以使用trimesh模块
转换为ply格式:
import trimesh
def to_ply(input_path, output_path, original_type):
mesh = trimesh.load(input_path, file_type=original_type) # read file
mesh.export(output_path, file_type=‘ply’) # convert to ply
- 1
- 2
- 3
- 4
- 5
2、调用上面定义的函数
to_ply('./test.obj', './test.ply', 'obj')
2.3 更高级的API
1、下面是更高级的API
from mmdet3d.apis import init_detector, inference_detector
config_file = ‘configs/votenet/votenet_8x8_scannet-3d-18class.py’
checkpoint_file = ‘my_checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth’
# build the model from a config file and a checkpoint file
model = init_detector(config_file, checkpoint_file, device=‘cuda:0’)
# test a single image and show the results
#point_cloud = ‘test.bin’
point_cloud = ‘./demo/data/kitti/kitti_000008.bin’
result, data = inference_detector(model, point_cloud)
# visualize the results and save the results in ‘results’ folder
model.show_results(data, result, out_dir=‘my_results’)
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
生成结果保存在my_results
目录下:
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ tree my_results/
my_results/
└── kitti_000008
├── kitti_000008_points.obj
└── kitti_000008_pred.obj
1 directory, 2 files
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
注意:
使用上面的代码在可视化的过程中,需要使用到open3d库
,因此需要先安装:
pip install open3d
2、显示结果
3 使用pcl可视化obj数据(这部分我自己写的)
首先,你需要确保你已经安装了pcl
(point cloud library),怎么安装pcl就自己百度吧!
1、下面是我写的一个简单可视化obj数据:pcl_viewer_obj.py
__Author__ = "Shliang"
__Email__ = "shliang0603@gmail.com"
import sys
import os
obj_name = sys.argv[1]
obj_file_abspath = os.path.abspath(obj_name)
obj_file_dir = os.path.dirname(obj_file_abspath)
output_pcd_name = obj_file_abspath.split(“/”)[-1].split(“.”)[0] + “.pcd”
output_pcd_file_abspath = os.path.join(obj_file_dir, output_pcd_name)
def pcl_obj2pcd(rm_pcd_file=True):
# 转换obj文件为pcd类型文件
cmd1 = "pcl_obj2pcd “ + obj_name + ” “ + output_pcd_file_abspath
print(f"cmd1: {
cmd1}”)
os.system(cmd1)
# 使用pcl_viewer可视化pcd文件
cmd2 = "pcl_viewer “ + output_pcd_file_abspath
print(f"cmd2: {
cmd2}”)
os.system(cmd2)
<span class="token comment"># 程序退出时,删除</span>
<span class="token keyword">if</span> rm_pcd_file<span class="token punctuation">:</span>
os<span class="token punctuation">.</span>remove<span class="token punctuation">(</span>output_pcd_file_abspath<span class="token punctuation">)</span>
sys<span class="token punctuation">.</span>exit<span class="token punctuation">(</span><span class="token punctuation">)</span>
if name == ‘main’:
pcl_obj2pcd(False)
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
2、然后执行命令:
python pcl_viewer_obj.py demo/kitti_000008/kitti_000008_points.obj
注意:
pcl_viewer更多参数:
- 先按h或H进入交互模式
- 按
g
可以显示网格 - 按
Shift+鼠标左键
:可以拖动点云图 - 按
j
截图当前的窗口,并保存为png图 - 按
c
显示当前 camera/window参数 - 按
e
退出交互界面
3、显示结果如下:
4 下载KITTI 3D object数据集和KITTI数据集详细介绍
4.1 下载KITTI 3D object数据集
1、数据集下载主页
- https://s3.eu-central-1.amazonaws.com/avg-kitti/
2、为了方便,这里我直接提供下载链接,大家直接点击即可下载,如果下载较慢,建议科学上网
!
-
1)Download left color images of object data set (12 GB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_image_2.zip
-
2)Download right color images, if you want to use stereo information (12 GB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_image_3.zip
-
3)Download the 3 temporally preceding frames (left color) (36 GB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_prev_2.zip
-
4)Download the 3 temporally preceding frames (right color) (36 GB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_prev_3.zip
-
5)Download Velodyne point clouds, if you want to use laser information (29 GB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_velodyne.zip
-
6)Download camera calibration matrices of object data set (16 MB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_calib.zip
-
7)Download training labels of object data set (5 MB) :https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_label_2.zip
-
8)Download object development kit (1 MB) :https://s3.eu-central-1.amazonaws.com/avg-kitti/devkit_object.zip
-
9)Download pre-trained LSVM baseline models (5 MB):https://s3.eu-central-1.amazonaws.com/avg-kitti/models_lsvm.zip
-
10)Download reference detections (L-SVM) for training and test set (800 MB):https://s3.eu-central-1.amazonaws.com/avg-kitti/data_object_det_2.zip
4.2 关于KITTI 3d数据集的详细介绍
如果你对KITTI 3d数据集还不是很了解,我建议你先去浏览一下面的几篇博客:
在开始之前,可以先对kitti的3D 数据集有一个更深入的了解与认识:
5 训练自己的数据集
这里我并没有训练自己的数据集,因为我手上并没有采集并标注好的激光雷达点云和摄像机图像数据,因此这里使用的是KITTI的3D object数据集
,官方提供了KITTI数据集格式的支持,如果你有自己的数据集,只需要把数据集转换成KITTI格式即可!!!
5.1 准备数据集和数据集的预处理
5.1.1 开始准备数据集
1、先创建数据目录:
mkdir -p data/kitti/ImageSets data/kitti/testing data/kitti/training
创建好如下:
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ tree data/kitti/
data/kitti/
├── ImageSets
├── testing
└── training
- 1
- 2
- 3
- 4
- 5
2、把数据集准备为kitti格式,如下所示是我下载的kitti的数据集(我这里是建立的动态链接,你们按照如下格式存放即可,参考):
训练和测试图片数量:
- trianing/image_2:
7481张
- testing/image_2:
7518张
3、其中,ImageSets自己创建,用于把数据集划分成训练、验证、测试
,把对应的txt文件复制到该位置即可:
可能有些小伙伴还是不知道如何划分这个数据集,不要慌,其实我也不知道
,但是,好在遇事不决,文档来协
,我们可以从官网文档查看到如何进行数据的具体划分
4、划分kitti数据集,其实官方已经提供了划分好的文件
wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/test.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/test.txt
wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/train.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/train.txt
wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/val.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/val.txt
wget -c https://raw.githubusercontent.com/traveller59/second.pytorch/master/second/data/ImageSets/trainval.txt --no-check-certificate --content-disposition -O ./data/kitti/ImageSets/trainval.txt
或者直接把上面的链接中对应的划分文件,下载到./data/kitti/ImageSets
目录下即可,下面是是链接,直接点击下载即可:
- test.txt:7518行
- train.txt:3712行
- val.txt:3769行
- trainval.txt:7481行
每个txt
文件中存储的都是文件名,例如:test.txt文件中的内容:
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d/data/kitti/ImageSets$ cat test.txt
000000
000001
000002
000003
000004
000005
......
007513
007514
007515
007516
007517
(base) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d/data/kitti/ImageSets$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
5.1.2 数据集的预处理
1、数据的预处理
python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
处理的时间比较久,大概需要半个小时,耐心等待吧!
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$ python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
Generate info. this may take several minutes.
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3712/3712, 18.2 task/s, elapsed: 205s, ETA: 0s
Kitti info train file is saved to data/kitti/kitti_infos_train.pkl
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 18.4 task/s, elapsed: 205s, ETA: 0s
Kitti info val file is saved to data/kitti/kitti_infos_val.pkl
Kitti info trainval file is saved to data/kitti/kitti_infos_trainval.pkl
Kitti info test file is saved to data/kitti/kitti_infos_test.pkl
create reduced point cloud for training set
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3712/3712, 18.7 task/s, elapsed: 199s, ETA: 0s
create reduced point cloud for validation set
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 18.8 task/s, elapsed: 200s, ETA: 0s
create reduced point cloud for testing set
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 7518/7518, 18.5 task/s, elapsed: 406s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3712/3712, 38.2 task/s, elapsed: 97s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 40.3 task/s, elapsed: 93s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 7481/7481, 39.5 task/s, elapsed: 189s, ETA: 0s
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 7518/7518, 42.4 task/s, elapsed: 177s, ETA: 0s
Create GT Database of KittiDataset
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 3712/3712, 17.3 task/s, elapsed: 215s, ETA: 0s
load 2207 Pedestrian database infos
load 14357 Car database infos
load 734 Cyclist database infos
load 1297 Van database infos
load 488 Truck database infos
load 224 Tram database infos
load 337 Misc database infos
load 56 Person_sitting database infos
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
2、数据预处理完之后,会生成一些pkl文件
,保存在./data/kitti目录下
:
5.1.3 数据预处理阶段可能会遇到的错误
1、可能报错:TypeError: expected dtype object, got 'numpy.dtype[float64]'
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
Generate info. this may take several minutes.
[ ] 0/3712, elapsed: 0s, ETA:points_v type: <class 'numpy.ndarray'>
Traceback (most recent call last):
File "tools/create_data.py", line 243, in <module>
out_dir=args.out_dir)
File "tools/create_data.py", line 23, in kitti_data_prep
kitti.create_kitti_info_file(root_path, info_prefix)
File "/home/shl/shl_res/MMlab/mmdetection3d/tools/data_converter/kitti_converter.py", line 118, in create_kitti_info_file
_calculate_num_points_in_gt(data_path, kitti_infos_train, relative_path)
File "/home/shl/shl_res/MMlab/mmdetection3d/tools/data_converter/kitti_converter.py", line 66, in _calculate_num_points_in_gt
points_v, rect, Trv2c, P2, image_info['image_shape'])
File "/home/shl/shl_res/MMlab/mmdetection3d/mmdet3d/core/bbox/box_np_ops.py", line 639, in remove_outside_points
frustum_surfaces = corner_to_surfaces_3d_jit(frustum[np.newaxis, ...])
TypeError: expected dtype object, got 'numpy.dtype[float64]'
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
我的版本:
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ conda list mmcv
# packages in environment at /home/shl/anaconda3/envs/open-mmlab:
#
# Name Version Build Channel
mmcv-full 1.3.1 pypi_0 pypi
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ conda list mmdet
# packages in environment at /home/shl/anaconda3/envs/open-mmlab:
#
# Name Version Build Channel
mmdet 2.11.0 dev_0 <develop>
mmdet3d 0.12.0 dev_0 <develop>
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ conda list numba
# packages in environment at /home/shl/anaconda3/envs/open-mmlab:
#
# Name Version Build Channel
numba 0.48.0 pypi_0 pypi
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ conda list numpy
# packages in environment at /home/shl/anaconda3/envs/open-mmlab:
#
# Name Version Build Channel
numpy 1.20.2 pypi_0 pypi
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
-
这个报错原因,就是因为
numba
的版本和numpy
版本不匹配导致的! -
对于numba==0.48.0要求对应的numpy版本:
1.15.0 <numpy<1.20.0
2、改成1.17.0版本后报错:ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
(1.18.0、1.19.0也是报这个错误)
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$ python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti
Traceback (most recent call last):
File "tools/create_data.py", line 5, in <module>
from tools.data_converter import kitti_converter as kitti
File "/home/shl/shl_res/MMlab/mmdetection3d/tools/data_converter/kitti_converter.py", line 7, in <module>
from mmdet3d.core.bbox import box_np_ops
File "/home/shl/shl_res/MMlab/mmdetection3d/mmdet3d/core/__init__.py", line 1, in <module>
from .anchor import * # noqa: F401, F403
File "/home/shl/shl_res/MMlab/mmdetection3d/mmdet3d/core/anchor/__init__.py", line 1, in <module>
from mmdet.core.anchor import build_anchor_generator
File "/home/shl/shl_res/MMlab/mmdetection/mmdet/core/__init__.py", line 5, in <module>
from .mask import * # noqa: F401, F403
File "/home/shl/shl_res/MMlab/mmdetection/mmdet/core/mask/__init__.py", line 2, in <module>
from .structures import BaseInstanceMasks, BitmapMasks, PolygonMasks
File "/home/shl/shl_res/MMlab/mmdetection/mmdet/core/mask/structures.py", line 6, in <module>
import pycocotools.mask as maskUtils
File "/home/shl/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/pycocotools-2.0.2-py3.7-linux-x86_64.egg/pycocotools/mask.py", line 3, in <module>
import pycocotools._mask as _mask
File "pycocotools/_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
(open-mmlab) shl@zhihui-mint:~/shl_res/MMlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
3、最终解决方式,是需要重新编译(mmcv等库包都是要重新编译的)
,把之前编译的文件全部删除之后再编译,其实我的最终解决方式(参考):
- 删除之前conda创建的虚拟环境,然后重型创建一个虚拟环境,重新安装环境(因为,我之前有下载很多库包,所以我不建议你也这样做)
- 首先安装numpy==1.18.0
- 然后在安装编译其他的库包
- mmcv安装版本为:
1.13.2
5.2 开始训练
5.2.1 开始训练
数据准备好了之后,就可以开始训练啦!
首先来看下,训练
都有哪些可选参数
:
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$ python tools/train.py -h
usage: train.py [-h] [--work-dir WORK_DIR] [--resume-from RESUME_FROM]
[--no-validate]
[--gpus GPUS | --gpu-ids GPU_IDS [GPU_IDS ...]] [--seed SEED]
[--deterministic] [--options OPTIONS [OPTIONS ...]]
[--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]]
[--launcher {none,pytorch,slurm,mpi}]
[--local_rank LOCAL_RANK] [--autoscale-lr]
config
Train a detector
positional arguments:
config train config file path
optional arguments:
-h, --help show this help message and exit
--work-dir WORK_DIR the dir to save logs and models
--resume-from RESUME_FROM
the checkpoint file to resume from
--no-validate whether not to evaluate the checkpoint during training
--gpus GPUS number of gpus to use (only applicable to non-
distributed training)
--gpu-ids GPU_IDS [GPU_IDS ...]
ids of gpus to use (only applicable to non-distributed
training)
--seed SEED random seed
--deterministic whether to set deterministic options for CUDNN
backend.
--options OPTIONS [OPTIONS ...]
override some settings in the used config, the key-
value pair in xxx=yyy format will be merged into
config file (deprecate), change to --cfg-options
instead.
--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]
override some settings in the used config, the key-
value pair in xxx=yyy format will be merged into
config file. If the value to be overwritten is a list,
it should be like key=“[a,b]” or key=a,b It also
allows nested list/tuple values, e.g.
key=“[(a,b),(c,d)]” Note that the quotation marks are
necessary and that no white space is allowed.
--launcher {
none,pytorch,slurm,mpi}
job launcher
--local_rank LOCAL_RANK
--autoscale-lr automatically scale lr with the number of gpus
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
2、根据自己的需求,选择相应的模型配置文件
,然后开始训练
python tools/train.py configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py
5.2.2 训练可能会遇到的问题:显存泄漏 RuntimeError: CUDA out of memory
首先说明一下我的显卡型号:
- NVIDIA GeForce
GTX 1080
- 显存大小:
8GB
1、在训练的时候报显存泄漏错误:
2021-04-29 10:07:13,961 - mmdet - INFO - load 337 Misc database infos
2021-04-29 10:07:13,961 - mmdet - INFO - load 56 Person_sitting database infos
2021-04-29 10:07:30,144 - mmdet - INFO - Start running, host: shl@zhihui-mint, work_dir: /home/shl/shl_res/mmlab/mmdetection3d/work_dirs/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class
2021-04-29 10:07:30,144 - mmdet - INFO - workflow: [('train', 1)], max: 80 epochs
Traceback (most recent call last):
File "tools/train.py", line 212, in <module>
main()
File "tools/train.py", line 208, in main
meta=meta)
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmdet/apis/train.py", line 170, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 51, in train
self.call_hook('after_train_iter')
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
getattr(hook, fn_name)(self)
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 35, in after_train_iter
runner.outputs['loss'].backward()
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/shl/anaconda3/envs/mmlab/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 472.00 MiB (GPU 0; 7.93 GiB total capacity; 5.03 GiB already allocated; 405.12 MiB free; 5.71 GiB reserved in total by PyTorch)
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$ ls
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
2、错误解决的方式
我的解决方式是通过修改batch_size大小
,但是我并没有在训练参数和配置文件中找到batch_size
的设置,因此我在github上提交了issues,原来batch_size
的参数是叫samples_per_gpu
,它的默认值是6
,我改成2即可正确运行了:
修改:
configs/_base_/datasets/kitti-3d-3class.py
第99行的sample_per_gpu=2
,大家根据自己显卡的显存进行修改
3、修改之后可以正确训练了
(mmlab) shl@zhihui-mint:~/shl_res/mmlab/mmdetection3d$ ./5_train_pointpillars_kitti_3class
2021-04-29 14:06:05,920 - mmdet - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce GTX 1080
CUDA_HOME: /usr/local/cuda-10.2
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.2
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
- CuDNN 7.6.5
- Magma 2.5.2
- Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.8.0
OpenCV: 4.5.1
MMCV: 1.3.2
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.2
MMDetection: 2.11.0
MMDetection3D: 0.12.0+86ef23d
------------------------------------------------------------
2021-04-29 14:06:06,763 - mmdet - INFO - Distributed training: False
2021-04-29 14:06:07,539 - mmdet - INFO - Config:
voxel_size = [0.16, 0.16, 4]
model = dict(
type=‘VoxelNet’,
voxel_layer=dict(
max_num_points=32,
point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1],
......
(pfn_layers): ModuleList(
(0): PFNLayer(
(norm): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(linear): Linear(in_features=9, out_features=64, bias=False)
)
)
)
(middle_encoder): PointPillarsScatter()
)
2021-04-29 14:06:07,828 - mmdet - INFO - load 2207 Pedestrian database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 14357 Car database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 734 Cyclist database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 1297 Van database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 488 Truck database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 224 Tram database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 337 Misc database infos
2021-04-29 14:06:07,828 - mmdet - INFO - load 56 Person_sitting database infos
2021-04-29 14:06:07,854 - mmdet - INFO - After filter database:
2021-04-29 14:06:07,854 - mmdet - INFO - load 2089 Pedestrian database infos
2021-04-29 14:06:07,854 - mmdet - INFO - load 13509 Car database infos
2021-04-29 14:06:07,854 - mmdet - INFO - load 684 Cyclist database infos
2021-04-29 14:06:07,855 - mmdet - INFO - load 1297 Van database infos
2021-04-29 14:06:07,855 - mmdet - INFO - load 488 Truck database infos
2021-04-29 14:06:07,855 - mmdet - INFO - load 224 Tram database infos
2021-04-29 14:06:07,855 - mmdet - INFO - load 337 Misc database infos
2021-04-29 14:06:07,855 - mmdet - INFO - load 56 Person_sitting database infos
2021-04-29 14:06:10,480 - mmdet - INFO - Start running, host: shl@zhihui-mint, work_dir: /home/shl/shl_res/mmlab/mmdetection3d/work_dirs/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class
2021-04-29 14:06:10,481 - mmdet - INFO - workflow: [(‘train’, 1)], max: 80 epochs
2021-04-29 14:06:33,687 - mmdet - INFO - Epoch [1][50/3712] lr: 1.000e-03, eta: 1 day, 12:30:51, time: 0.443, data_time: 0.188, memory: 1982, loss_cls: 0.7894, loss_bbox: 1.8513, loss_dir: 0.1458, loss: 2.7864, grad_norm: 12.9730
2021-04-29 14:06:45,074 - mmdet - INFO - Epoch [1][100/3712] lr: 1.000e-03, eta: 1 day, 3:38:37, time: 0.228, data_time: 0.002, memory: 1982, loss_cls: 0.6169, loss_bbox: 1.3952, loss_dir: 0.1387, loss: 2.1508, grad_norm: 10.1342
2021-04-29 14:06:56,451 - mmdet - INFO - Epoch [1][150/3712] lr: 1.000e-03, eta: 1 day, 0:40:45, time: 0.228, data_time: 0.002, memory: 1982, loss_cls: 0.5477, loss_bbox: 1.2741, loss_dir: 0.1361, loss: 1.9578, grad_norm: 9.0859
2021-04-29 14:07:08,080 - mmdet - INFO - Epoch [1][200/3712] lr: 1.000e-03, eta: 23:17:57, time: 0.233, data_time: 0.002, memory: 1982, loss_cls: 0.4903, loss_bbox: 1.1924, loss_dir: 0.1352, loss: 1.8180, grad_norm: 9.9547
2021-04-29 14:07:19,705 - mmdet - INFO - Epoch [1][250/3712] lr: 1.000e-03, eta: 22:28:06, time: 0.232, data_time: 0.002, memory: 1982, loss_cls: 0.4968, loss_bbox: 1.1553, loss_dir: 0.1337, loss: 1.7858, grad_norm: 8.9025
2021-04-29 14:07:31,165 - mmdet - INFO - Epoch [1][300/3712] lr: 1.000e-03, eta: 21:52:06, time: 0.229, data_time: 0.002, memory: 1988, loss_cls: 0.4640, loss_bbox: 1.1292, loss_dir: 0.1308, loss: 1.7241, grad_norm: 8.6915
2021-04-29 14:07:42,547 - mmdet - INFO - Epoch [1][350/3712] lr: 1.000e-03, eta: 21:25:14, time: 0.228, data_time: 0.002, memory: 1988, loss_cls: 0.4549, loss_bbox: 1.1149, loss_dir: 0.1269, loss: 1.6966, grad_norm: 8.3001
2021-04-29 14:07:53,968 - mmdet - INFO - Epoch [1][400/3712] lr: 1.000e-03, eta: 21:05:31, time: 0.228, data_time: 0.002, memory: 1989, loss_cls: 0.4386, loss_bbox: 1.0360, loss_dir: 0.1252, loss: 1.5998, grad_norm: 8.4704
2021-04-29 14:08:05,299 - mmdet - INFO - Epoch [1][450/3712] lr: 1.000e-03, eta: 20:49:09, time: 0.227, data_time: 0.002, memory: 1989, loss_cls: 0.4229, loss_bbox: 1.0270, loss_dir: 0.1207, loss: 1.5706, grad_norm: 7.9375
2021-04-29 14:08:16,598 - mmdet - INFO - Epoch [1][500/3712] lr: 1.000e-03, eta: 20:35:42, time: 0.226, data_time: 0.002, memory: 1999, loss_cls: 0.3957, loss_bbox: 0.9792, loss_dir: 0.1196, loss: 1.4945, grad_norm: 7.5102
2021-04-29 14:08:27,929 - mmdet - INFO - Epoch [1][550/3712] lr: 1.000e-03, eta: 20:24:57, time: 0.227, data_time: 0.002, memory: 1999, loss_cls: 0.3900, loss_bbox: 0.9470, loss_dir: 0.1154, loss: 1.4523, grad_norm: 7.0491
2021-04-29 14:08:39,170 - mmdet - INFO - Epoch [1][600/3712] lr: 1.001e-03, eta: 20:15:13, time: 0.225, data_time: 0.002, memory: 1999, loss_cls: 0.3822, loss_bbox: 0.9552, loss_dir: 0.1166, loss: 1.4541, grad_norm: 7.0271
2021-04-29 14:08:50,525 - mmdet - INFO - Epoch [1][650/3712] lr: 1.001e-03, eta: 20:07:49, time: 0.227, data_time: 0.002, memory: 1999, loss_cls: 0.3699, loss_bbox: 0.9163, loss_dir: 0.1167, loss: 1.4029, grad_norm: 7.1026
2021-04-29 14:09:01,906 - mmdet - INFO - Epoch [1][700/3712] lr: 1.001e-03, eta: 20:01:38, time: 0.228, data_time: 0.002, memory: 1999, loss_cls: 0.3610, loss_bbox: 0.9148, loss_dir: 0.1121, loss: 1.3880, grad_norm: 7.1760
2021-04-29 14:09:13,433 - mmdet - INFO - Epoch [1][750/3712] lr: 1.001e-03, eta: 19:57:12, time: 0.231, data_time: 0.002, memory: 1999, loss_cls: 0.3580, loss_bbox: 0.9246, loss_dir: 0.1131, loss: 1.3957, grad_norm: 7.2312
2021-04-29 14:09:24,840 - mmdet - INFO - Epoch [1][800/3712] lr: 1.001e-03, eta: 19:52:34, time: 0.228, data_time: 0.002, memory: 1999, loss_cls: 0.3579, loss_bbox: 0.8869, loss_dir: 0.1129, loss: 1.3578, grad_norm: 6.2828
2021-04-29 14:09:36,273 - mmdet - INFO - Epoch [1][850/3712] lr: 1.001e-03, eta: 19:48:37, time: 0.229, data_time: 0.002, memory: 1999, loss_cls: 0.3360, loss_bbox: 0.8947, loss_dir: 0.1105, loss: 1.3412, grad_norm: 7.1239
2021-04-29 14:09:47,786 - mmdet - INFO - Epoch [1][900/3712] lr: 1.001e-03, eta: 19:45:30, time: 0.230, data_time: 0.002, memory: 1999, loss_cls: 0.3390, loss_bbox: 0.8851, loss_dir: 0.1093, loss: 1.3334, grad_norm: 6.7943
2021-04-29 14:09:59,886 - mmdet - INFO - Epoch [1][950/3712] lr: 1.001e-03, eta: 19:45:45, time: 0.242, data_time: 0.003, memory: 1999, loss_cls: 0.3461, loss_bbox: 0.8913, loss_dir: 0.1123, loss: 1.3497, grad_norm: 6.1542
2021-04-29 14:10:11,571 - mmdet - INFO - Exp name: hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py
2021-04-29 14:10:11,571 - mmdet - INFO - Epoch [1][1000/3712] lr: 1.002e-03, eta: 19:43:55, time: 0.234, data_time: 0.002, memory: 1999, loss_cls: 0.3369, loss_bbox: 0.8745, loss_dir: 0.1111, loss: 1.3225, grad_norm: 6.1611
2021-04-29 14:10:23,420 - mmdet - INFO - Epoch [1][1050/3712] lr: 1.002e-03, eta: 19:43:00, time: 0.237, data_time: 0.003, memory: 1999, loss_cls: 0.3332, loss_bbox: 0.8765, loss_dir: 0.1118, loss: 1.3216, grad_norm: 6.1433
2021-04-29 14:10:34,900 - mmdet - INFO - Epoch [1][1100/3712] lr: 1.002e-03, eta: 19:40:30, time: 0.230, data_time: 0.002, memory: 1999, loss_cls: 0.3238, loss_bbox: 0.8262, loss_dir: 0.1109, loss: 1.2610, grad_norm: 6.0072
2021-04-29 14:10:46,469 - mmdet - INFO - Epoch [1][1150/3712] lr: 1.002e-03, eta: 19:38:35, time: 0.231, data_time: 0.002, memory: 1999, loss_cls: 0.3230, loss_bbox: 0.8378, loss_dir: 0.1102, loss: 1.2709, grad_norm: 6.3352
2021-04-29 14:10:58,090 - mmdet - INFO - Epoch [1][1200/3712] lr: 1.002e-03, eta: 19:37:01, time: 0.232, data_time: 0.002, memory: 1999, loss_cls: 0.3116, loss_bbox: 0.8198, loss_dir: 0.1110, loss: 1.2423, grad_norm: 5.8147
2021-04-29 14:11:09,553 - mmdet - INFO - Epoch [1][1250/3712] lr: 1.002e-03, eta: 19:34:56, time: 0.229, data_time: 0.002, memory: 1999, loss_cls: 0.3150, loss_bbox: 0.8013, loss_dir: 0.1077, loss: 1.2240, grad_norm: 5.8795
2021-04-29 14:11:21,253 - mmdet - INFO - Epoch [1][1300/3712] lr: 1.003e-03, eta: 19:33:54, time: 0.234, data_time: 0.002, memory: 1999, loss_cls: 0.3214, loss_bbox: 0.8256, loss_dir: 0.1113, loss: 1.2583, grad_norm: 5.7060
......
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
4、显存使用:nvidia-smi -lms
^C(mmlab) shl@zhihui-mint:~/shl_res$ nvidia-smi
Thu Apr 29 14:18:47 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:01:00.0 On | N/A |
| 42% 62C P2 106W / 198W | 4401MiB / 8116MiB | 87% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+------------------------------------<span class="token o