Hugging 的 Trainer 训练多输入网络遇到的问题 The batch received was empty, your model won‘t be able to train on it

背景描述

使用 Hugging 的 Trainer 训练自定义网络，自定义网络包括多个输入，如下图所示：

代码详见 HuggingFace 自定义数据集，使用 Trainer 训练，多个输入，定义多流网络

错误描述

root@55esp9ftlj3h2-0:/codes/ssm-gaze-estimation/swin_v2_headpose# python3 train.py 
/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py:436: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches']). Please pass an `accelerate.DataLoaderConfiguration` instead: 
dataloader_config = DataLoaderConfiguration(dispatch_batches=None)
  warnings.warn(
Detected kernel version 3.10.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
  0%|                                                                                                                                                          | 0/12622110 [00:00<?, ?it/s]/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4065: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/torch/nn/functional.py:4003: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
Traceback (most recent call last):
  File "train.py", line 57, in <module>
    trainer.train()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1556, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1838, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2686, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2636, in _prepare_inputs
    raise ValueError(
ValueError: The batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: inputs,label,label_ids.
  0%|          | 0/12622110 [00:01<?, ?it/s] 

问题分析

Hugging 的 Trainer 在训练过程中会自动删除未被使用的列（指的是在模型的前向传播中未被使用），结果导致把所有列都删除了，从而导致 The batch received was empty。

问题解决

在训练参数中指定不要移除无用列 remove_unused_columns=False。

training_args = TrainingArguments(output_dir=args.outdir,
	     num_train_epochs=args.epoch,
	     per_device_train_batch_size=args.batchSize,
	     per_device_eval_batch_size=args.batchSize,
	     logging_dir=args.loggingdir, 
	     logging_strategy='epoch',
	     save_strategy='epoch',
	     evaluation_strategy='epoch',
	     # 不要删除无用的列
	     remove_unused_columns=False)