Torch学习笔记四

Torchvision

导入tochvision

import torchvision

torchvision.dataset: 

下载CIFAR-10数据集

train_set = torchvision.datasets.CIFAR10(root="./dataset", download=True)
test_set = torchvision.datasets.CIFAR10(root="./dataset", train=False, download=True)

download=True可以获得下载地址，也可以在存在压缩包的情况下进行解压

数据集图片类型：

(<PIL.Image.Image image mode=RGB size=32x32 at 0x1E8898222C0>, 3)

可以看出图片类型为Image img，mode为RGB，大小为32*32，通道数为3

数据集结构：

The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

 代码一：

import torchvision
from torchvision import transforms
from PIL import Image

train_set = torchvision.datasets.CIFAR10(root="./dataset", download=True)
test_set = torchvision.datasets.CIFAR10(root="./dataset", train=False, download=True)

img = test_set[5000][0]  # type:Image.Image
label = test_set[5000][1]
print("第一张图数据类型：", test_set[0])
print("label:", test_set.classes[label])
img.show()

该代码调用测试集第5001张图片，获取PIL类型图片和标签，输出结果如下：

Files already downloaded and verified
Files already downloaded and verified
第一张图数据类型： (<PIL.Image.Image image mode=RGB size=32x32 at 0x25EE83AA4D0>, 3)
label: horse

 

 代码二：

import torchvision
from torchvision import transforms
from torch.utils.tensorboard import SummaryWriter
from PIL import Image

trans_compose = transforms.Compose(
    [
        transforms.ToTensor()
    ]
)
train_set = torchvision.datasets.CIFAR10(root="./dataset", transform=trans_compose, download=True)
test_set = torchvision.datasets.CIFAR10(root="./dataset", transform=trans_compose, train=False, download=True)

writer = SummaryWriter("CIFAR10_logs")
for i in range(100):
    img, label = test_set[i]
    print(f"第{i}张图的label:", test_set.classes[label])
    writer.add_image("test_set", img, i)
writer.close()

通过trans_compose对数据集图像进行处理，这里只是转变为tensor类型图片，体现在数据集的transform的属性设置为trans_compose

Dataloader:

导入dataloader库：

from torch.utils.data import DataLoader

Dataloader的注释：

Data loader. Combines a dataset and a sampler, and provides an iterable over
the given dataset.

The :class:`~torch.utils.data.DataLoader` supports both map-style and
iterable-style datasets with single- or multi-process loading, customizing
loading order and optional automatic batching (collation) and memory pinning.

See :py:mod:`torch.utils.data` documentation page for more details.

Args:
    dataset (Dataset): dataset from which to load the data.
    batch_size (int, optional): how many samples per batch to load
        (default: ``1``).
    shuffle (bool, optional): set to ``True`` to have the data reshuffled
        at every epoch (default: ``False``).
    sampler (Sampler or Iterable, optional): defines the strategy to draw
        samples from the dataset. Can be any ``Iterable`` with ``__len__``
        implemented. If specified, :attr:`shuffle` must not be specified.
    batch_sampler (Sampler or Iterable, optional): like :attr:`sampler`, but
        returns a batch of indices at a time. Mutually exclusive with
        :attr:`batch_size`, :attr:`shuffle`, :attr:`sampler`,
        and :attr:`drop_last`.
    num_workers (int, optional): how many subprocesses to use for data
        loading. ``0`` means that the data will be loaded in the main process.
        (default: ``0``)
    collate_fn (Callable, optional): merges a list of samples to form a
        mini-batch of Tensor(s).  Used when using batched loading from a
        map-style dataset.
    pin_memory (bool, optional): If ``True``, the data loader will copy Tensors
        into device/CUDA pinned memory before returning them.  If your data elements
        are a custom type, or your :attr:`collate_fn` returns a batch that is a custom type,
        see the example below.
    drop_last (bool, optional): set to ``True`` to drop the last incomplete batch,
        if the dataset size is not divisible by the batch size. If ``False`` and
        the size of dataset is not divisible by the batch size, then the last batch
        will be smaller. (default: ``False``)
    timeout (numeric, optional): if positive, the timeout value for collecting a batch
        from workers. Should always be non-negative. (default: ``0``)
    worker_init_fn (Callable, optional): If not ``None``, this will be called on each
        worker subprocess with the worker id (an int in ``[0, num_workers - 1]``) as
        input, after seeding and before data loading. (default: ``None``)
    generator (torch.Generator, optional): If not ``None``, this RNG will be used
        by RandomSampler to generate random indexes and multiprocessing to generate
        `base_seed` for workers. (default: ``None``)
    prefetch_factor (int, optional, keyword-only arg): Number of batches loaded
        in advance by each worker. ``2`` means there will be a total of
        2 * num_workers batches prefetched across all workers. (default value depends
        on the set value for num_workers. If value of num_workers=0 default is ``None``.
        Otherwise if value of num_workers>0 default is ``2``).
    persistent_workers (bool, optional): If ``True``, the data loader will not shutdown
        the worker processes after a dataset has been consumed once. This allows to
        maintain the workers `Dataset` instances alive. (default: ``False``)
    pin_memory_device (str, optional): the data loader will copy Tensors
        into device pinned memory before returning them if pin_memory is set to true.

参数解释：

1.  dataset：torchvision.Dataset数据集
2. batch_size：一批次多少图，这里不是将图片融合为一张图片，例如单张tensor图片的shape是（3,32,32）那么batch_size=64返回的img的shape是（64,3,32,32）
3. shuffle：洗牌，每次Dataloader的结果不同
4. drop_last:batch_size除不尽，则丢掉除不尽的照片

导入测试集：

test_loader = DataLoader(dataset=test_set, batch_size=64, shuffle=True, drop_last=True)

综合代码如下：

from torch.utils.data import DataLoader
import torchvision
from torchvision import transforms
from torch.utils.tensorboard import SummaryWriter

trans_compose = transforms.Compose(
    [
        transforms.ToTensor()
    ]
)
test_set = torchvision.datasets.CIFAR10(root="dataset", train=False, transform=trans_compose)
test_loader = DataLoader(dataset=test_set, batch_size=64, shuffle=True, drop_last=True)

writer = SummaryWriter("CIFAR10_logs")
for epoch in range(2):
    count = 0
    for data in test_loader:
        img, _ = data
        writer.add_images(f"Epoch{epoch}", img, global_step=count)
        count += 1

print("结束")