huggingface transformers loadset 导入本地文件

点击查看 Huggingface详细入门介绍之dataset库

loadset 导入本地文件

import os

from datasets import load_dataset

data_home = r"D:\数据集路径"
#
data_dict = {
    "train": os.path.join(data_home, "train.json"),
    "test": os.path.join(data_home, "test.json"),
}
datasets = load_dataset("json", data_files=data_dict)
print(datasets)
print(datasets["train"][0])

load_dataset("json", data_files=data_dict)

json : 表示导入的本地文件是 json文件