MindSpore自定义数据集训练时遇到错误ValueError: The data pipeline is not a tree

问题描述

def process_dataset(dataset, processor, max_seq_len=128, batch_size=16, shuffle=False):
  def process_image(image):
    image = Image.fromarray(image.astype('uint8'), 'L')
    image = image.convert("RGB")
    pixel_values = processor(image, return_tensors="ms").pixel_values
    return pixel_values
  dataset = dataset.map(operations=process_image, input_columns="image", output_columns='image')
  dataset = dataset.map(operations=text_tokenizer, input_columns="text", output_columns="text")
  dataset = dataset.batch(batch_size)
  return dataset
train_dataset = process_dataset(iam["train"],processor=processor)
for batch in train_dataset1.create_dict_iterator():
  print(batch['image'].dtype)

iam["train"]为mindspore.dataset.engine.datasets_user_defined.GeneratorDataset

运行时报错

ValueError Traceback (most recent call last) in <cell line: 1>() ----> 1 for batch in train_dataset1.create_dict_iterator(): 2 print(batch['image'].dtype) 3 break

/usr/local/lib/python3.10/dist-packages/mindspore/dataset/engine/datasets.py in parse_tree(self, getter_mode) 422 """ 423 if len(self.parent) > 1: --> 424 raise ValueError("The data pipeline is not a tree (i.e., one node has 2 consumers)") 425 ir_children = [d.parse_tree(getter_mode) for d in self.children] 426 # Bootstrap can only be performed on a copy of the original dataset node.

ValueError: The data pipeline is not a tree (i.e., one node has 2 consumers)

解决方案

train_dataset = process_dataset(iam["train"],processor=processor) 这行代码申明创建的是train_dataset，而下面for循环中使用的是train_dataset1.