错误信息
[10/19 20:29:19] d2.data.datasets.coco INFO: Loaded 38743 images in COCO format from /home/dlsvr3/server/train_server/uploads/物料纸箱20221019/out_dir/COCO/annotations/train.json
[10/19 20:29:20] d2.data.build INFO: Removed 0 images with no usable annotations. 38743 images left.
[10/19 20:29:20] d2.data.dataset_mapper INFO: [DatasetMapper] Augmentations used in training: [ResizeShortestEdge(short_edge_length=(640,), max_size=640, sample_style='choice'), RandomFlip()]
[10/19 20:29:20] d2.data.build INFO: Using training sampler TrainingSampler
[10/19 20:29:20] d2.data.common INFO: Serializing 38743 elements to byte tensors and concatenating them all ...
[10/19 20:29:20] d2.data.common INFO: Serialized dataset takes 101.06 MiB
[10/19 20:29:20] fvcore.common.checkpoint INFO: [Checkpointer] Loading from mm01_p/models/model_final_f10217.pkl ...
[10/19 20:29:20] fvcore.common.checkpoint INFO: Reading a file from 'Detectron2 Model Zoo'
[10/19 20:29:20] d2.engine.train_loop INFO: Starting training from iteration 0
[10/19 20:29:38] d2.engine.train_loop ERROR: Exception during training:
Traceback (most recent call last):
File "/home/dlsvr3/detectron2/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/home/dlsvr3/detectron2/detectron2/engine/defaults.py", line 494, in run_step
self._trainer.run_step()
File "/home/dlsvr3/detectron2/detectron2/engine/train_loop.py", line 273, in run_step
loss_dict = self.model(data)
File "/home/dlsvr3/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dlsvr3/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 163, in forward
_, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
File "/home/dlsvr3/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dlsvr3/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 739, in forward
losses = self._forward_box(features, proposals)
File "/home/dlsvr3/detectron2/detectron2/modeling/roi_heads/roi_heads.py", line 804, in _forward_box
losses = self.box_predictor.losses(predictions, proposals)
File "/home/dlsvr3/detectron2/detectron2/modeling/roi_heads/fast_rcnn.py", line 324, in losses
proposal_boxes, gt_boxes, proposal_deltas, gt_classes
File "/home/dlsvr3/detectron2/detectron2/modeling/roi_heads/fast_rcnn.py", line 338, in box_reg_loss
fg_inds = nonzero_tuple((gt_classes >= 0) & (gt_classes < self.num_classes))[0]
File "/home/dlsvr3/detectron2/detectron2/layers/wrappers.py", line 132, in nonzero_tuple
return x.nonzero(as_tuple=True)
RuntimeError: CUDA error: device-side assert triggered
[10/19 20:29:38] d2.engine.hooks INFO: Total training time: 0:00:18 (0:00:00 on hooks)
[10/19 20:29:38] d2.utils.events INFO: iter: 0 lr: N/A max_mem: 2963M
这个错误出现在使用detectron2训练mask rcnn
出现原因
cfg.MODEL.RETINANET.NUM_CLASSES = class_name_number + 1
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = class_name_number + 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = class_name_number + 1
训练的类别个数有100+个类别,上面的几个NUM_CLASSES 没有设置正确或者只设置了其中的一个,而默认值是80,与训练集的类别个数对应不上,所以出现了错误。
解决办法
解决方法也很简单,以下都设置正确就好了,类别数+1是加了背景类别
cfg.MODEL.RETINANET.NUM_CLASSES = class_name_number + 1
cfg.MODEL.SEM_SEG_HEAD.NUM_CLASSES = class_name_number + 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = class_name_number + 1