数据集加载之ImageFolder
ImageFolder
一个通用的数据加载器,数据集中的数据以以下方式组织
函数如下
ImageFolder(root, transform``=``None``, target_transform``=``None``, loader``=``default_loader)
参数解释
-
root 指定路径加载图片
-
transform:对PIL Image进行的转换操作,transform的输入是使用loader读取图片的返回对象
-
target_transform:对label的转换
-
loader:给定路径后如何读取图片,默认读取为RGB格式的PIL Image对象
label是按照文件夹名顺序排序后存成字典,即{类名:类序号(从0开始)},一般来说最好直接将文件夹命名为从0开始的数字,这样会和ImageFolder
实际的label
一致,如果不是这种命名规范,建议看看self.class_to_idx
属性以了解label和文件夹名的映射关系
可见分成了cat和dog两类
import torchvision.datasets as dset
dataset = dset.ImageFolder('./data/dogcat_2') #没有transform,先看看取得的原始图像数据
print(dataset.classes) #根据分的文件夹的名字来确定的类别
print(dataset.class_to_idx) #按顺序为这些类别定义索引为0,1...
print(dataset.imgs) #返回从所有文件夹中得到的图片的路径以及其类别
返回
['cat', 'dog']
{'cat': 0, 'dog': 1}
[('./data/dogcat_2/cat/cat.12484.jpg', 0), ('./data/dogcat_2/cat/cat.12485.jpg', 0), ('./data/dogcat_2/cat/cat.12486.jpg', 0), ('./data/dogcat_2/cat/cat.12487.jpg', 0), ('./data/dogcat_2/dog/dog.12496.jpg', 1), ('./data/dogcat_2/dog/dog.12497.jpg', 1), ('./data/dogcat_2/dog/dog.12498.jpg', 1), ('./data/dogcat_2/dog/dog.12499.jpg', 1)]
查看得到的图片数据:
#从返回结果可见得到的数据仍是PIL Image对象
print(dataset[0])
print(dataset[0][0])
print(dataset[0][1]) #得到的是类别0,即cat
返回:
(<PIL.Image.Image image mode=RGB size=497x500 at 0x11D99A9B0>, 0)
<PIL.Image.Image image mode=RGB size=497x500 at 0x11DD24278>
0
再通过transform
变成张量
data_transform = {
"train": transforms.Compose([transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ToTensor()
]),
"val": transforms.Compose([transforms.Resize((224, 224)), # cannot 224, must (224, 224)
transforms.ToTensor()
])}
train_dataset = datasets.ImageFolder(root=os.path.join(data_root, "train"),
transform=data_transform["train"])
print(train_dataset[0])
print(train_dataset[0][0])
print(train_dataset[0][1]) #得到的是类别0,即cat
得到
(tensor([[[0.3765, 0.3765, 0.3765, ..., 0.3765, 0.3765, 0.3765],
[0.3765, 0.3765, 0.3765, ..., 0.3765, 0.3765, 0.3765],
[0.3804, 0.3804, 0.3804, ..., 0.3765, 0.3765, 0.3765],
...,
[0.4941, 0.4941, 0.4902, ..., 0.3804, 0.3765, 0.3765],
[0.5098, 0.5098, 0.5059, ..., 0.3804, 0.3765, 0.3765],
[0.5098, 0.5098, 0.5059, ..., 0.3804, 0.3765, 0.3765]],
[[0.5686, 0.5686, 0.5686, ..., 0.5843, 0.5804, 0.5804],
[0.5686, 0.5686, 0.5686, ..., 0.5843, 0.5804, 0.5804],
[0.5725, 0.5725, 0.5725, ..., 0.5843, 0.5804, 0.5804],
...,
[0.5686, 0.5686, 0.5686, ..., 0.5961, 0.5922, 0.5922],
[0.5686, 0.5686, 0.5686, ..., 0.5961, 0.5922, 0.5922],
[0.5686, 0.5686, 0.5686, ..., 0.5961, 0.5922, 0.5922]],
[[0.4824, 0.4824, 0.4824, ..., 0.4902, 0.4902, 0.4902],
[0.4824, 0.4824, 0.4824, ..., 0.4902, 0.4902, 0.4902],
[0.4863, 0.4863, 0.4863, ..., 0.4902, 0.4902, 0.4902],
...,
[0.3647, 0.3686, 0.3804, ..., 0.4824, 0.4784, 0.4784],
[0.3451, 0.3490, 0.3608, ..., 0.4824, 0.4784, 0.4784],
[0.3451, 0.3490, 0.3608, ..., 0.4824, 0.4784, 0.4784]]]), 0)
省略
图像Tensor [........]
标签Tensor [0]
可以知道train_dataset[0]
是一个tuple
里面一个是张量一个是标量
print(train_dataset[0][0].shape)
print(train_dataset[0][1])
返回
torch.Size([3, 224, 224])
0
DataLoader产生批训练数据
for epoch in range(epoch_num):
for image, label in train_loader:
print("label: ", labels, labels.dtype)
print("imges: ", images, images.dtype)
返回
label: tensor([0]) torch.int64
imges: tensor([[[[0.2275, 0.2275, 0.2235, ..., 0.2196, 0.2196, 0.2196],
[0.2275, 0.2275, 0.2235, ..., 0.2196, 0.2196, 0.2196],
[0.2235, 0.2235, 0.2235, ..., 0.2157, 0.2157, 0.2157],
...,
[0.2392, 0.2392, 0.2392, ..., 0.2235, 0.2235, 0.2235],
[0.2353, 0.2353, 0.2353, ..., 0.2235, 0.2275, 0.2275],
[0.2353, 0.2353, 0.2353, ..., 0.2235, 0.2275, 0.2275]],
[[0.2392, 0.2392, 0.2353, ..., 0.2275, 0.2275, 0.2275],
[0.2392, 0.2392, 0.2353, ..., 0.2275, 0.2275, 0.2275],
[0.2353, 0.2353, 0.2353, ..., 0.2235, 0.2235, 0.2235],
...,
[0.2275, 0.2275, 0.2314, ..., 0.2196, 0.2196, 0.2196],
[0.2235, 0.2235, 0.2275, ..., 0.2196, 0.2196, 0.2196],
[0.2235, 0.2235, 0.2275, ..., 0.2196, 0.2196, 0.2196]],
[[0.1961, 0.1961, 0.1961, ..., 0.1765, 0.1765, 0.1765],
[0.1961, 0.1961, 0.1961, ..., 0.1765, 0.1765, 0.1765],
[0.1922, 0.1922, 0.1922, ..., 0.1725, 0.1725, 0.1725],
...,
[0.1529, 0.1529, 0.1529, ..., 0.1686, 0.1686, 0.1686],
[0.1490, 0.1490, 0.1490, ..., 0.1686, 0.1686, 0.1686],
[0.1490, 0.1490, 0.1490, ..., 0.1686, 0.1686, 0.1686]]]]) torch.float32
enumerate加载
enumerate() 函数用于将一个可遍历的数据对象(如列表、元组或字符串)组合为一个索引序列,同时列出数据和数据下标,一般用在 for 循环当中。
可以用这个循环加载batchsize
下面的每一元组
,来得到相应的图片及标签
多元素赋值
当Python检测到等号左边是多个变量,右边是list或者是tuple之后,会自动执行list和tuple的解压,将它依次赋值给对应的元素:
l = [1, 2]
a, b = l
所以可以
for step, data in enumerate(train_bar):
images, labels = data
最后nn.CrossEntropyLoss()计算loss回归
nn.CrossEntropyLoss()
是nn.logSoftmax()
和nn.NLLLoss()
的整合,可以直接使用它来替换网络中的这两个操作。下面我们来看一下计算过程。
其中x是网络的输出向量,class是真实标签
举个例子,一个三分类网络对某个输入样本的输出为[-0.7715, -0.6205,-0.2562],该样本的真实标签为0,则用nn.CrossEntropyLoss()
计算的损失为