ppyoloe问题

07/19 22:35:09 - mmengine - INFO - Epoch(train)  [4][ 60/580]  base_lr: 2.5000e-04 lr: 1.5509e-04  eta: 4 days, 0:
55:13  time: 6.2245  data_time: 0.4387  memory: 1211  loss: 4.0579  loss_cls: 0.9532  loss_state: 1.2636  loss_bbo
x: 0.3440  loss_dfl: 1.4971
07/19 22:36:52 - mmengine - INFO - Epoch(train)  [4][ 70/580]  base_lr: 2.5000e-04 lr: 1.5595e-04  eta: 4 days, 1:
04:14  time: 7.6080  data_time: 0.4719  memory: 5738  loss: 4.0051  loss_cls: 0.9280  loss_state: 1.2617  loss_bbo
x: 0.3368  loss_dfl: 1.4786
07/19 22:37:58 - mmengine - INFO - Epoch(train)  [4][ 80/580]  base_lr: 2.5000e-04 lr: 1.5681e-04  eta: 4 days, 0:
58:00  time: 8.1634  data_time: 0.0477  memory: 1211  loss: 4.0580  loss_cls: 0.9506  loss_state: 1.2706  loss_bbo
x: 0.3783  loss_dfl: 1.4584
07/19 22:39:41 - mmengine - INFO - Epoch(train)  [4][ 90/580]  base_lr: 2.5000e-04 lr: 1.5767e-04  eta: 4 days, 1:
06:24  time: 9.7714  data_time: 0.0519  memory: 1211  loss: 4.0740  loss_cls: 0.9612  loss_state: 1.2761  loss_bbo
x: 0.3790  loss_dfl: 1.4578
07/19 22:41:32 - mmengine - INFO - Epoch(train)  [4][100/580]  base_lr: 2.5000e-04 lr: 1.5853e-04  eta: 4 days, 1:
18:20  time: 10.1447  data_time: 0.0393  memory: 1299  loss: 4.0076  loss_cls: 0.9446  loss_state: 1.2327  loss_bb
ox: 0.3803  loss_dfl: 1.4501
07/19 22:43:03 - mmengine - INFO - Epoch(train)  [4][110/580]  base_lr: 2.5000e-04 lr: 1.5940e-04  eta: 4 days, 1:
21:53  time: 9.4774  data_time: 0.0465  memory: 1299  loss: 4.0742  loss_cls: 0.9750  loss_state: 1.2301  loss_bbo
x: 0.4177  loss_dfl: 1.4514
07/19 22:43:44 - mmengine - INFO - Epoch(train)  [4][120/580]  base_lr: 2.5000e-04 lr: 1.6026e-04  eta: 4 days, 1:
05:55  time: 8.2485  data_time: 0.0239  memory: 1123  loss: 4.2037  loss_cls: 1.0055  loss_state: 1.2765  loss_bbo
x: 0.4376  loss_dfl: 1.4840
07/19 22:44:31 - mmengine - INFO - Epoch(train)  [4][130/580]  base_lr: 2.5000e-04 lr: 1.6112e-04  eta: 4 days, 0:
52:08  time: 7.8639  data_time: 0.0401  memory: 1299  loss: 4.1204  loss_cls: 0.9918  loss_state: 1.2751  loss_bbo
x: 0.4120  loss_dfl: 1.4415
07/19 22:45:00 - mmengine - INFO - Epoch(train)  [4][140/580]  base_lr: 2.5000e-04 lr: 1.6198e-04  eta: 4 days, 0:
31:01  time: 6.3823  data_time: 0.0468  memory: 1299  loss: 4.1752  loss_cls: 1.0122  loss_state: 1.2948  loss_bbo
x: 0.4150  loss_dfl: 1.4531
07/19 22:45:53 - mmengine - INFO - Epoch(train)  [4][150/580]  base_lr: 2.5000e-04 lr: 1.6284e-04  eta: 4 days, 0:
20:00  time: 5.2226  data_time: 0.0467  memory: 1211  loss: 4.1133  loss_cls: 0.9960  loss_state: 1.2773  loss_bbo
x: 0.3973  loss_dfl: 1.4427
07/19 22:46:23 - mmengine - INFO - Epoch(train)  [4][160/580]  base_lr: 2.5000e-04 lr: 1.6371e-04  eta: 3 days, 23
:59:52  time: 4.0014  data_time: 0.0396  memory: 1299  loss: 4.1325  loss_cls: 0.9886  loss_state: 1.2972  loss_bb
ox: 0.3905  loss_dfl: 1.4561
07/19 22:47:54 - mmengine - INFO - Epoch(train)  [4][170/580]  base_lr: 2.5000e-04 lr: 1.6457e-04  eta: 4 days, 0:
04:03  time: 4.9978  data_time: 0.0289  memory: 1123  loss: 4.0423  loss_cls: 0.9726  loss_state: 1.2667  loss_bbo
x: 0.3852  loss_dfl: 1.4178
07/19 22:49:26 - mmengine - INFO - Epoch(train)  [4][180/580]  base_lr: 2.5000e-04 lr: 1.6543e-04  eta: 4 days, 0:
07:57  time: 5.8808  data_time: 0.0185  memory: 1299  loss: 4.1775  loss_cls: 1.0039  loss_state: 1.2959  loss_bbo
x: 0.3878  loss_dfl: 1.4899
07/19 22:49:50 - mmengine - INFO - Epoch(train)  [4][190/580]  base_lr: 2.5000e-04 lr: 1.6629e-04  eta: 3 days, 23
:46:10  time: 5.8069  data_time: 0.0079  memory: 1211  loss: 4.1025  loss_cls: 0.9986  loss_state: 1.2759  loss_bb
ox: 0.3874  loss_dfl: 1.4406
07/19 22:51:30 - mmengine - INFO - Epoch(train)  [4][200/580]  base_lr: 2.5000e-04 lr: 1.6716e-04  eta: 3 days, 23
:53:28  time: 6.7419  data_time: 0.0106  memory: 1299  loss: 4.0270  loss_cls: 0.9840  loss_state: 1.2463  loss_bb
ox: 0.3891  loss_dfl: 1.4076
07/19 23:02:37 - mmengine - INFO - Epoch(train)  [4][210/580]  base_lr: 2.5000e-04 lr: 1.6802e-04  eta: 4 days, 3:
35:58  time: 19.4827  data_time: 0.0172  memory: 1299  loss: 3.9365  loss_cls: 0.9562  loss_state: 1.1972  loss_bb
ox: 0.3909  loss_dfl: 1.3922
07/19 23:02:56 - mmengine - INFO - Epoch(train)  [4][220/580]  base_lr: 2.5000e-04 lr: 1.6888e-04  eta: 4 days, 3:
11:23  time: 18.0313  data_time: 0.0172  memory: 1123  loss: 3.9469  loss_cls: 0.9466  loss_state: 1.1939  loss_bb
ox: 0.3961  loss_dfl: 1.4104
07/19 23:04:27 - mmengine - INFO - Epoch(train)  [4][230/580]  base_lr: 2.5000e-04 lr: 1.6974e-04  eta: 4 days, 3:
14:05  time: 18.0290  data_time: 0.0515  memory: 1809  loss: 3.8146  loss_cls: 0.8968  loss_state: 1.1688  loss_bb
ox: 0.3726  loss_dfl: 1.3764
07/19 23:06:09 - mmengine - INFO - Epoch(train)  [4][240/580]  base_lr: 2.5000e-04 lr: 1.7060e-04  eta: 4 days, 3:
20:38  time: 19.5705  data_time: 0.0513  memory: 1299  loss: 3.8529  loss_cls: 0.8854  loss_state: 1.1818  loss_bb
ox: 0.3745  loss_dfl: 1.4112
07/19 23:07:30 - mmengine - INFO - Epoch(train)  [4][250/580]  base_lr: 2.5000e-04 lr: 1.7147e-04  eta: 4 days, 3:
19:33  time: 19.1939  data_time: 0.0699  memory: 1123  loss: 3.9280  loss_cls: 0.9007  loss_state: 1.2024  loss_bb
ox: 0.3729  loss_dfl: 1.4521
07/19 23:09:59 - mmengine - INFO - Exp name: ppyoloe_plus_s_fast_8xb8-80e_coco_20240719_184020
07/19 23:09:59 - mmengine - INFO - Epoch(train)  [4][260/580]  base_lr: 2.5000e-04 lr: 1.7233e-04  eta: 4 days, 3:
43:49  time: 8.8543  data_time: 0.0840  memory: 1299  loss: 4.0175  loss_cls: 0.9366  loss_state: 1.2415  loss_bbo
x: 0.3666  loss_dfl: 1.4728
07/19 23:10:29 - mmengine - INFO - Epoch(train)  [4][270/580]  base_lr: 2.5000e-04 lr: 1.7319e-04  eta: 4 days, 3:
23:24  time: 9.0525  data_time: 0.0855  memory: 1125  loss: 3.9124  loss_cls: 0.9142  loss_state: 1.1960  loss_bbo
x: 0.3407  loss_dfl: 1.4615
07/19 23:11:15 - mmengine - INFO - Epoch(train)  [4][280/580]  base_lr: 2.5000e-04 lr: 1.7405e-04  eta: 4 days, 3:
09:43  time: 8.1679  data_time: 0.0544  memory: 1299  loss: 3.9072  loss_cls: 0.9227  loss_state: 1.1882  loss_bbo
x: 0.3505  loss_dfl: 1.4458
07/19 23:12:09 - mmengine - INFO - Epoch(train)  [4][290/580]  base_lr: 2.5000e-04 lr: 1.7491e-04  eta: 4 days, 2:
58:39  time: 7.2122  data_time: 0.0544  memory: 1139  loss: 3.8454  loss_cls: 0.9033  loss_state: 1.1660  loss_bbo
x: 0.3427  loss_dfl: 1.4334
07/19 23:14:07 - mmengine - INFO - Epoch(train)  [4][300/580]  base_lr: 2.5000e-04 lr: 1.7578e-04  eta: 4 days, 3:
10:56  time: 7.9462  data_time: 0.0460  memory: 1299  loss: 3.7560  loss_cls: 0.8613  loss_state: 1.1528  loss_bbo
x: 0.3260  loss_dfl: 1.4159
07/19 23:16:20 - mmengine - INFO - Epoch(train)  [4][310/580]  base_lr: 2.5000e-04 lr: 1.7664e-04  eta: 4 days, 3:
28:22  time: 7.6028  data_time: 0.0254  memory: 1211  loss: 3.7026  loss_cls: 0.8481  loss_state: 1.1537  loss_bbo
x: 0.3134  loss_dfl: 1.3874
07/19 23:16:56 - mmengine - INFO - Epoch(train)  [4][320/580]  base_lr: 2.5000e-04 lr: 1.7750e-04  eta: 4 days, 3:
10:56  time: 7.7392  data_time: 0.0238  memory: 1299  loss: 3.7546  loss_cls: 0.8488  loss_state: 1.1982  loss_bbo
x: 0.3319  loss_dfl: 1.3758
07/19 23:18:29 - mmengine - INFO - Epoch(train)  [4][330/580]  base_lr: 2.5000e-04 lr: 1.7836e-04  eta: 4 days, 3:
14:21  time: 8.6788  data_time: 0.0149  memory: 963  loss: 3.8670  loss_cls: 0.8823  loss_state: 1.2236  loss_bbox
: 0.3470  loss_dfl: 1.4140
07/19 23:19:29 - mmengine - INFO - Epoch(train)  [4][340/580]  base_lr: 2.5000e-04 lr: 1.7922e-04  eta: 4 days, 3:
05:36  time: 8.7994  data_time: 0.0149  memory: 1044  loss: 3.8786  loss_cls: 0.8796  loss_state: 1.2530  loss_bbo
x: 0.3386  loss_dfl: 1.4073
07/19 23:22:00 - mmengine - INFO - Epoch(train)  [4][350/580]  base_lr: 2.5000e-04 lr: 1.8009e-04  eta: 4 days, 3:
29:13  time: 9.4643  data_time: 0.0053  memory: 1299  loss: 4.1052  loss_cls: 0.9432  loss_state: 1.3361  loss_bbo
x: 0.3719  loss_dfl: 1.4541
07/19 23:22:51 - mmengine - INFO - Epoch(train)  [4][360/580]  base_lr: 2.5000e-04 lr: 1.8095e-04  eta: 4 days, 3:
17:08  time: 7.8188  data_time: 0.0102  memory: 963  loss: 4.0853  loss_cls: 0.9371  loss_state: 1.3109  loss_bbox
: 0.3897  loss_dfl: 1.4476
07/19 23:25:08 - mmengine - INFO - Epoch(train)  [4][370/580]  base_lr: 2.5000e-04 lr: 1.8181e-04  eta: 4 days, 3:
35:30  time: 9.8407  data_time: 0.0102  memory: 1299  loss: 4.1099  loss_cls: 0.9707  loss_state: 1.3064  loss_bbo
x: 0.3791  loss_dfl: 1.4537
07/19 23:26:01 - mmengine - INFO - Epoch(train)  [4][380/580]  base_lr: 2.5000e-04 lr: 1.8267e-04  eta: 4 days, 3:
24:44  time: 9.0422  data_time: 0.0134  memory: 1419  loss: 3.9697  loss_cls: 0.9292  loss_state: 1.2594  loss_bbo
x: 0.3538  loss_dfl: 1.4273
[E ProcessGroupNCCL.cpp:587] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BAS
E, Timeout(ms)=1800000) ran for 1807354 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BAS
E, Timeout(ms)=1800000) ran for 1807440 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BAS
E, Timeout(ms)=1800000) ran for 1807458 milliseconds before timing out.
Traceback (most recent call last):
  File "./tools/train.py", line 126, in <module>
Traceback (most recent call last):
  File "./tools/train.py", line 126, in <module>
    main()
  File "./tools/train.py", line 122, in main
    runner.train()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1746, in train
    main()
  File "./tools/train.py", line 122, in main
    runner.train()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1746, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    model = self.train_loop.run()  # type: ignore
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_iter(idx, data_batch)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    self.run_epoch()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    outputs = self.runner.model.train_step(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 121
, in train_step
    self.run_iter(idx, data_batch)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    losses = self._run_forward(data, mode='loss')
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 161
, in _run_forward
    outputs = self.runner.model.train_step(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 121
, in train_step
    results = self(**data, mode=mode)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    losses = self._run_forward(data, mode='loss')
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 161
, in _run_forward
    results = self(**data, mode=mode)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in f
orward
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in f
orward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/base.py", line 92, in forward
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/base.py", line 92, in forward
    return self.loss(inputs, data_samples)
      File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 77, in loss
return self.loss(inputs, data_samples)
  File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 77, in loss
    x = self.extract_feat(batch_inputs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 157, in extract_feat
    x = self.extract_feat(batch_inputs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 157, in extract_feat
    x = self.backbone(batch_inputs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    x = self.backbone(batch_inputs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/backbones/base_backbone.py", line 221, in forward
    x = layer(x)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/backbones/base_backbone.py", line 221, in forward
        x = layer(x)return forward_call(*input, **kwargs)

  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
        input = module(input)return forward_call(*input, **kwargs)

  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
    input = module(input)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1295, in forward
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1295, in forward
    y2 = self.blocks(self.conv2(x))
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    y2 = self.blocks(self.conv2(x))
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
    input = module(input)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
    input = module(input)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1154, in forward
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1154, in forward
    y = self.conv2(y)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 252, in forward
        y = self.conv2(y)self.rbr_dense(inputs) +

  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmcv/cnn/bricks/conv_module.py", line 281, in for
ward
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 252, in forward
    self.rbr_dense(inputs) +
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    x = self.norm(x)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 749, in forw
ard
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmcv/cnn/bricks/conv_module.py", line 281, in for
ward
    x = self.norm(x)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 749, in forw
ard
    return sync_batch_norm.apply(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/_functions.py", line 42, in forw
ard
    return sync_batch_norm.apply(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/_functions.py", line 42, in forw
ard
    dist._all_gather_base(combined_flat, combined, process_group, async_op=False)
    dist._all_gather_base(combined_flat, combined, process_group, async_op=False)  File "/home/lsw/miniconda3/envs
/mi/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2070, in _all_gather_base

  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2070
, in _all_gather_base
    work = group._allgather_base(output_tensor, input_tensor)
RuntimeError: NCCL communicator was aborted on rank 0.  Original reason for failure was: [Rank 0] Watchdog caught 
collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BASE, Timeout(ms)=1800000) ran for 1807354 milliseconds b
efore timing out.
    work = group._allgather_base(output_tensor, input_tensor)
RuntimeError: NCCL communicator was aborted on rank 3.  Original reason for failure was: [Rank 3] Watchdog caught 
collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BASE, Timeout(ms)=1800000) ran for 1807440 milliseconds b
efore timing out.
Traceback (most recent call last):
  File "./tools/train.py", line 126, in <module>
    main()
  File "./tools/train.py", line 122, in main
    runner.train()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1746, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    self.run_iter(idx, data_batch)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    outputs = self.runner.model.train_step(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 121
, in train_step
    losses = self._run_forward(data, mode='loss')
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/wrappers/distributed.py", line 161
, in _run_forward
    results = self(**data, mode=mode)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 886, in f
orward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/base.py", line 92, in forward
    return self.loss(inputs, data_samples)
  File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 77, in loss
    x = self.extract_feat(batch_inputs)
  File "/home/lsw/hl/mi/mmdet/models/detectors/single_stage.py", line 157, in extract_feat
    x = self.backbone(batch_inputs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/backbones/base_backbone.py", line 221, in forward
    x = layer(x)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
    input = module(input)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1295, in forward
    y2 = self.blocks(self.conv2(x))
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forw
ard
    input = module(input)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 1154, in forward
    y = self.conv2(y)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/hl/mi/mmyolo/models/layers/yolo_bricks.py", line 252, in forward
    self.rbr_dense(inputs) +
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmcv/cnn/bricks/conv_module.py", line 281, in for
ward
    x = self.norm(x)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_
impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/batchnorm.py", line 749, in forw
ard
    return sync_batch_norm.apply(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/_functions.py", line 42, in forw
ard
    dist._all_gather_base(combined_flat, combined, process_group, async_op=False)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2070
, in _all_gather_base
    work = group._allgather_base(output_tensor, input_tensor)
RuntimeError: NCCL communicator was aborted on rank 2.  Original reason for failure was: [Rank 2] Watchdog caught 
collective operation timeout: WorkNCCL(OpType=_ALLGATHER_BASE, Timeout(ms)=1800000) ran for 1807458 milliseconds b
efore timing out.
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA
 kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are ta
king the entire process down.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 13725 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 13724) of binary: /ho
me/lsw/miniconda3/envs/mi/bin/python
Traceback (most recent call last):
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <modul
e>
    main()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in 
__call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in 
launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./tools/train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-07-19_23:57:21
  host      : lswPlus
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 13726)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-07-19_23:57:21
  host      : lswPlus
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 13727)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 13727
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-07-19_23:57:21
  host      : lswPlus
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 13724)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

dist.init_process_group(backend='nccl', init_method='env://', timeout=datetime.timedelta(seconds=5400))

07/20 17:37:28 - mmengine - INFO - Epoch(train)  [3][2460/4634]  base_lr: 1.2500e-04 lr: 6.3266e-05  eta: 15:32:14  time: 0.1281  data_time: 0.0223  memory: 654  loss: 4.5739  loss_cls: 1.0988  loss_state: 1.4604  loss_bbox: 0.6126  loss_dfl: 1.4021
Traceback (most recent call last):
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 990, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/queue.py", line 175, in get
    while not self._qsize():
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/queue.py", line 209, in _qsize
    return len(self.queue)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 8104) is killed by signal: Killed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lsw/HL/othercode/mi/tools/train.py", line 125, in <module>
    main()
  File "/home/lsw/HL/othercode/mi/tools/train.py", line 121, in main
    runner.train()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1746, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 111, in run_epoch
    for idx, data_batch in enumerate(self.dataloader):
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1186, in _next_data
    idx, data = self._get_data()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1142, in _get_data
    success, data = self._try_get_data()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1003, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 8104) exited unexpectedly

07/20 21:31:35 - mmengine - INFO - Epoch(train)  [1][ 260/4634]  base_lr: 1.2500e-04 lr: 1.3973e-06  eta: 14:31:26  time: 0.1365  data_time: 0.0374  memory: 715  loss: 7.9384  loss_cls: 2.2571  loss_state: 3.1435  loss_bbox: 0.7062  loss_dfl: 1.8315
Traceback (most recent call last):
  File "/home/lsw/HL/othercode/mi/tools/train.py", line 125, in <module>
    main()
  File "/home/lsw/HL/othercode/mi/tools/train.py", line 121, in main
    runner.train()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1746, in train
    model = self.train_loop.run()  # type: ignore
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 96, in run
    self.run_epoch()
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 112, in run_epoch
    self.run_iter(idx, data_batch)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/runner/loops.py", line 128, in run_iter
    outputs = self.runner.model.train_step(
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/mmengine/model/base_model/base_model.py", line 113, in train_step
    data = self.data_preprocessor(data, True)
  File "/home/lsw/miniconda3/envs/mi/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/lsw/HL/othercode/mi/mmyolo/models/data_preprocessors/data_preprocessor.py", line 163, in forward
    _input = _input[[2, 1, 0], ...]
RuntimeError: CUDA out of memory. Tried to allocate 5.77 GiB (GPU 0; 11.91 GiB total capacity; 5.89 GiB already allocated; 4.86 GiB free; 5.98 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

代码

_backend_args = None
_multiscale_resize_transforms = [
    dict(
        transforms=[
            dict(scale=(
                640,
                640,
            ), type='YOLOv5KeepRatioResize'),
            dict(
                allow_scale_up=False,
                pad_val=dict(img=114),
                scale=(
                    640,
                    640,
                ),
                type='LetterResize'),
        ],
        type='Compose'),
    dict(
        transforms=[
            dict(scale=(
                320,
                320,
            ), type='YOLOv5KeepRatioResize'),
            dict(
                allow_scale_up=False,
                pad_val=dict(img=114),
                scale=(
                    320,
                    320,
                ),
                type='LetterResize'),
        ],
        type='Compose'),
    dict(
        transforms=[
            dict(scale=(
                960,
                960,
            ), type='YOLOv5KeepRatioResize'),
            dict(
                allow_scale_up=False,
                pad_val=dict(img=114),
                scale=(
                    960,
                    960,
                ),
                type='LetterResize'),
        ],
        type='Compose'),
]
backend_args = None
# base_lr = 0.001
base_lr = 0.000125
custom_hooks = [
    dict(
        ema_type='ExpMomentumEMA',
        momentum=0.0002,
        priority=49,
        strict_load=False,
        type='EMAHook',
        update_buffers=True),
]
data_root = 'data/coco/'
dataset_type = 'YOLOv5CocoDataset'
deepen_factor = 0.33
default_hooks = dict(
    checkpoint=dict(
        interval=5,
        max_keep_ckpts=16,
        save_best='auto',
        type='CheckpointHook'),
    logger=dict(interval=10, type='LoggerHook'),
    param_scheduler=dict(
        min_lr_ratio=0.0,
        start_factor=0.0,
        total_epochs=96,
        type='PPYOLOEParamSchedulerHook',
        warmup_epochs=5,
        warmup_min_iter=1000),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(type='mmdet.DetVisualizationHook'))
default_scope = 'mmyolo'
env_cfg = dict(
    cudnn_benchmark=False,
    dist_cfg=dict(backend='nccl'),
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
img_scale = (
    640,
    640,
)
img_scales = [
    (
        640,
        640,
    ),
    (
        320,
        320,
    ),
    (
        960,
        960,
    ),
]
launcher = 'none'
load_from = '/home/lsw/HL/othercode/mi/checkpoints/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 80
metainfo = dict(
    classes=(
        'humanchild',
        'humanpregnant_woman',
        'humanold',
        'humanhandicapped',
    ),
    palette=[
        (
            20,
            220,
            60,
        ),
    ])
model = dict(
    backbone=dict(
        act_cfg=dict(inplace=True, type='SiLU'),
        attention_cfg=dict(
            act_cfg=dict(type='HSigmoid'), type='EffectiveSELayer'),
        block_cfg=dict(
            shortcut=True, type='PPYOLOEBasicBlock', use_alpha=True),
        deepen_factor=0.33,
        norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
        type='PPYOLOECSPResNet',
        use_large_stem=True,
        widen_factor=0.5),
    bbox_head=dict(
        bbox_coder=dict(type='DistancePointBBoxCoder'),
        head_module=dict(
            act_cfg=dict(inplace=True, type='SiLU'),
            featmap_strides=[
                8,
                16,
                32,
            ],
            in_channels=[
                192,
                384,
                768,
            ],
            norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
            num_base_priors=1,
            num_classes=4,
            num_state=4,
            reg_max=16,
            type='PPYOLOEHeadModule',
            widen_factor=0.5),
        loss_bbox=dict(
            bbox_format='xyxy',
            iou_mode='giou',
            loss_weight=2.5,
            reduction='mean',
            return_iou=False,
            type='IoULoss'),
        loss_cls=dict(
            alpha=0.75,
            gamma=2.0,
            iou_weighted=True,
            loss_weight=1.0,
            reduction='sum',
            type='mmdet.VarifocalLoss',
            use_sigmoid=True),
        loss_dfl=dict(
            loss_weight=0.125,
            reduction='mean',
            type='mmdet.DistributionFocalLoss'),
        loss_state=dict(
            alpha=0.75,
            gamma=2.0,
            iou_weighted=True,
            loss_weight=1.0,
            reduction='sum',
            type='mmdet.VarifocalLoss',
            use_sigmoid=True),
        prior_generator=dict(
            offset=0.5, strides=[
                8,
                16,
                32,
            ], type='mmdet.MlvlPointGenerator'),
        type='PPYOLOEHead'),
    data_preprocessor=dict(
        batch_augments=[
            dict(
                interval=1,
                keep_ratio=False,
                random_interp=True,
                random_size_range=(
                    320,
                    800,
                ),
                size_divisor=32,
                type='PPYOLOEBatchRandomResize'),
        ],
        bgr_to_rgb=True,
        mean=[
            0.0,
            0.0,
            0.0,
        ],
        pad_size_divisor=32,
        std=[
            255.0,
            255.0,
            255.0,
        ],
        type='PPYOLOEDetDataPreprocessor'),
    neck=dict(
        act_cfg=dict(inplace=True, type='SiLU'),
        block_cfg=dict(
            shortcut=False, type='PPYOLOEBasicBlock', use_alpha=False),
        deepen_factor=0.33,
        drop_block_cfg=None,
        in_channels=[
            256,
            512,
            1024,
        ],
        norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
        num_blocks_per_layer=3,
        num_csplayer=1,
        out_channels=[
            192,
            384,
            768,
        ],
        type='PPYOLOECSPPAFPN',
        use_spp=True,
        widen_factor=0.5),
    test_cfg=dict(
        max_per_img=300,
        multi_label=True,
        nms=dict(iou_threshold=0.7, type='nms'),
        nms_pre=1000,
        score_thr=0.01),
    train_cfg=dict(
        assigner=dict(
            alpha=1,
            beta=6,
            eps=1e-09,
            num_classes=4,
            topk=13,
            type='mi_BatchTaskAlignedAssigner'),
        initial_assigner=dict(
            iou_calculator=dict(type='mmdet.BboxOverlaps2D'),
            num_classes=4,
            topk=9,
            type='BatchATSSAssigner'),
        initial_epoch=30),
    type='YOLODetector')
num_classes = 4
num_state = 4
optim_wrapper = dict(
    optimizer=dict(
        lr=0.000125,
        momentum=0.9,
        nesterov=False,
        type='SGD',
        weight_decay=0.0005),
    paramwise_cfg=dict(norm_decay_mult=0.0),
    type='OptimWrapper')
param_scheduler = None
persistent_workers = True
resume = False
save_epoch_intervals = 80
strides = [
    8,
    16,
    32,
]
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
    batch_size=1,
    dataset=dict(
        ann_file='annotations/instances_val2017.json',
        data_prefix=dict(img='images/val2017/'),
        data_root='data/coco/',
        filter_cfg=dict(filter_empty_gt=True, min_size=0),
        metainfo=dict(
            classes=(
                'humanchild',
                'humanpregnant_woman',
                'humanold',
                'humanhandicapped',
            ),
            palette=[
                (
                    20,
                    220,
                    60,
                ),
            ]),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(
                height=640,
                interpolation='bicubic',
                keep_ratio=False,
                type='mmdet.FixShapeResize',
                width=640),
            dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='mmdet.PackDetInputs'),
        ],
        test_mode=True,
        type='YOLOv5CocoDataset'),
    drop_last=False,
    num_workers=1,
    persistent_workers=True,
    pin_memory=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
    ann_file='data/coco/annotations/instances_val2017.json',
    metric='bbox',
    proposal_nums=(
        100,
        1,
        10,
    ),
    type='mmdet.CocoMetric')
test_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(
        height=640,
        interpolation='bicubic',
        keep_ratio=False,
        type='mmdet.FixShapeResize',
        width=640),
    dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'scale_factor',
        ),
        type='mmdet.PackDetInputs'),
]
train_batch_size_per_gpu = 1
train_cfg = dict(max_epochs=80, type='EpochBasedTrainLoop', val_interval=80)
train_dataloader = dict(
    batch_size=1,
    collate_fn=dict(type='yolov5_collate', use_ms_training=True),
    dataset=dict(
        ann_file='annotations/instances_train2017.json',
        data_prefix=dict(img='images/train2017/'),
        data_root='data/coco/',
        filter_cfg=dict(filter_empty_gt=True, min_size=0),
        metainfo=dict(
            classes=(
                'humanchild',
                'humanpregnant_woman',
                'humanold',
                'humanhandicapped',
            ),
            palette=[
                (
                    20,
                    220,
                    60,
                ),
            ]),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(type='PPYOLOERandomDistort'),
            dict(mean=(
                103.53,
                116.28,
                123.675,
            ), type='mmdet.Expand'),
            dict(type='PPYOLOERandomCrop'),
            dict(prob=0.5, type='mmdet.RandomFlip'),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'flip',
                    'flip_direction',
                ),
                type='mmdet.PackDetInputs'),
        ],
        type='YOLOv5CocoDataset'),
    num_workers=1,
    persistent_workers=True,
    pin_memory=True,
    sampler=dict(shuffle=True, type='DefaultSampler'))
train_num_workers = 1
train_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='PPYOLOERandomDistort'),
    dict(mean=(
        103.53,
        116.28,
        123.675,
    ), type='mmdet.Expand'),
    dict(type='PPYOLOERandomCrop'),
    dict(prob=0.5, type='mmdet.RandomFlip'),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'flip',
            'flip_direction',
        ),
        type='mmdet.PackDetInputs'),
]
tta_model = dict(
    tta_cfg=dict(max_per_img=300, nms=dict(iou_threshold=0.65, type='nms')),
    type='mmdet.DetTTAModel')
tta_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(
        transforms=[
            [
                dict(
                    transforms=[
                        dict(scale=(
                            640,
                            640,
                        ), type='YOLOv5KeepRatioResize'),
                        dict(
                            allow_scale_up=False,
                            pad_val=dict(img=114),
                            scale=(
                                640,
                                640,
                            ),
                            type='LetterResize'),
                    ],
                    type='Compose'),
                dict(
                    transforms=[
                        dict(scale=(
                            320,
                            320,
                        ), type='YOLOv5KeepRatioResize'),
                        dict(
                            allow_scale_up=False,
                            pad_val=dict(img=114),
                            scale=(
                                320,
                                320,
                            ),
                            type='LetterResize'),
                    ],
                    type='Compose'),
                dict(
                    transforms=[
                        dict(scale=(
                            960,
                            960,
                        ), type='YOLOv5KeepRatioResize'),
                        dict(
                            allow_scale_up=False,
                            pad_val=dict(img=114),
                            scale=(
                                960,
                                960,
                            ),
                            type='LetterResize'),
                    ],
                    type='Compose'),
            ],
            [
                dict(prob=1.0, type='mmdet.RandomFlip'),
                dict(prob=0.0, type='mmdet.RandomFlip'),
            ],
            [
                dict(type='mmdet.LoadAnnotations', with_bbox=True),
            ],
            [
                dict(
                    meta_keys=(
                        'img_id',
                        'img_path',
                        'ori_shape',
                        'img_shape',
                        'scale_factor',
                        'pad_param',
                        'flip',
                        'flip_direction',
                    ),
                    type='mmdet.PackDetInputs'),
            ],
        ],
        type='TestTimeAug'),
]
val_batch_size_per_gpu = 1
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
    batch_size=1,
    dataset=dict(
        ann_file='annotations/instances_val2017.json',
        data_prefix=dict(img='images/val2017/'),
        data_root='data/coco/',
        filter_cfg=dict(filter_empty_gt=True, min_size=0),
        metainfo=dict(
            classes=(
                'humanchild',
                'humanpregnant_woman',
                'humanold',
                'humanhandicapped',
            ),
            palette=[
                (
                    20,
                    220,
                    60,
                ),
            ]),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(
                height=640,
                interpolation='bicubic',
                keep_ratio=False,
                type='mmdet.FixShapeResize',
                width=640),
            dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='mmdet.PackDetInputs'),
        ],
        test_mode=True,
        type='YOLOv5CocoDataset'),
    drop_last=False,
    num_workers=1,
    persistent_workers=True,
    pin_memory=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
    ann_file='data/coco/annotations/instances_val2017.json',
    metric='bbox',
    proposal_nums=(
        100,
        1,
        10,
    ),
    type='mmdet.CocoMetric')
val_num_workers = 1
vis_backends = [
    dict(type='LocalVisBackend'),
]
visualizer = dict(
    name='visualizer',
    type='mmdet.DetLocalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
    ])
widen_factor = 0.5
work_dir = 'HL'