深度学习数据集

1 huggingface datasets

需要先安装 datasets库

pip install datasets

用coco数据集举例，我们可以搜索coco，然后通过页面右侧的use this dataset或者是 clone respository来获取数据集

https://huggingface.co/datasets/phiyodr/coco2017

huggingface的dataset数据类型

from datasets import load_dataset

ds = load_dataset("phiyodr/coco2017")
print(ds)

Generating train split: 100%|██████████| 118287/118287 [00:00<00:00, 1413307.31 examples/s]
Generating validation split: 100%|██████████| 5000/5000 [00:00<00:00, 1229064.06 examples/s]
DatasetDict({
    train: Dataset({
        features: ['license', 'file_name', 'coco_url', 'height', 'width', 'date_captured', 'flickr_url', 'image_id', 'ids', 'captions'],
        num_rows: 118287
    })
    validation: Dataset({
        features: ['license', 'file_name', 'coco_url', 'height', 'width', 'date_captured', 'flickr_url', 'image_id', 'ids', 'captions'],
        num_rows: 5000
    })
})

可以通过上述地址看到,coco数据集用.arrow格式储存了

pandas dataFrame格式

import pandas as pd

splits = {'train': 'data/train-00000-of-00001-0084e041f1902997.parquet', 'validation': 'data/validation-00000-of-00001-e3c37e369512a3aa.parquet'}
df = pd.read_parquet("hf://datasets/phiyodr/coco2017/" + splits["train"])
print(df)

git下载

右侧点击 clone repository

2 kaggle datasets

Find Open Datasets and Machine Learning Projects | KaggleDownload Open Datasets on 1000s of Projects + Share Projects on One Platform. Explore Popular Topics Like Government, Sports, Medicine, Fintech, Food, More. Flexible Data Ingestion.https://www.kaggle.com/datasets

3 通过迅雷下载

需要自行获取数据集下载地址

coco

COCO2017 训练数据：http://images.cocodataset.org/zips/train2017.zip
http://images.cocodataset.org/annotations/annotations_trainval2017.zip
COCO2017验证数据：http://images.cocodataset.org/zips/val2017.zip
http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip
COCO2017测试数据集：http://images.cocodataset.org/zips/test2017.zip
http://images.cocodataset.org/annotations/image_info_test2017.zip

4 经典数据集介绍

WIT

任务：视觉 nlp多模态

huggingface

https://huggingface.co/datasets/google/withttps://huggingface.co/datasets/google/witgithub

GitHub - google-research-datasets/wit: WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages.WIT (Wikipedia-based Image Text) Dataset is a large multimodal multilingual dataset comprising 37M+ image-text sets with 11M+ unique images across 100+ languages. - google-research-datasets/withttps://github.com/google-research-datasets/wit数据集论文地址

https://arxiv.org/pdf/2103.01913https://arxiv.org/pdf/2103.01913

dogs vs cats

任务：图像二分类

下载地址：

Dogs vs. Cats Redux: Kernels Edition | Kaggle