python-torch如何保存数据集，以yelp_review_full数据集为例

方法一

先自动下载，然后使用save_to_disk保存到本地

dataset = load_dataset("yelp_review_full")
dataset.save_to_disk('./yelp_review_full_disk')

使用的时候如下操作即可

dataset=datasets.load_from_disk("./csdn/ddset/yelp_review_full")

方法二

先自动下载

dataset = load_dataset("yelp_review_full")

执行完了以后，会在路径/home/xxxuser/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf中保存

-rw-r----- 1 1user ma-group       1786 Aug 18 14:29 dataset_info.json
-rw-r----- 1 1user ma-group   37282608 Aug 18 14:29 yelp_review_full-test.arrow
-rw-r----- 1 1user ma-group  483954656 Aug 18 14:29 yelp_review_full-train.arrow

上面这是三个文件就是我们需要的，可以使用下面方法使用数据集

from torch.utils.data import Dataset, DataLoader
import pyarrow as pa

class YelpReviewFullDataset(Dataset):
    def __init__(self, file_path):
        self.data = pa.parquet.read_table(file_path).to_pandas()

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data.iloc[idx]

train_dataset = YelpReviewFullDataset('./csdn/ddset/yelp_review_full/yelp_review_full-train.arrow')
test_dataset = YelpReviewFullDataset('./csdn/ddset/yelp_review_full/yelp_review_full-test.arrow')

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)