ResNet笔记 Deep Residual Learning for Image Recognition

---

论文基本信息

作者：Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun

关键词：CNN、ResNet

领域：CV

链接：https://arxiv.org/pdf/1512.03385.pdf

参考：

• 【ResNet论文逐段精读【论文精读】】 https://www.bilibili.com/video/BV1P3411y7nn/?share_source=copy_web&vd_source=0ac66901e8dd624384e51529673d725a

• 【【精读AI论文】ResNet深度残差网络】 https://www.bilibili.com/video/BV1vb4y1k7BV/?share_source=copy_web&vd_source=0ac66901e8dd624384e51529673d725a

• 【6.2 使用pytorch搭建ResNet并基于迁移学习训练】https://www.bilibili.com/video/BV14E411H7Uw/?spm_id_from=333.337.search-card.all.click&vd_source=20c37d260c9535497db1af49e1fd2cbd

• 深度学习经典论文分析（六）-Deep Residual Learning for Image Recognition https://zhuanlan.zhihu.com/p/430442777

---

解决的问题

深层的网络很难去训练，会有“退化”现象产生。残差块使得很深的网络更加容易训练，甚至可以训练一千层的网络。

残差网络对随后的深层神经网络设计产生了深远的影响，无论是卷积类网络还是全连接网络。

---

Deep Residual Networks：

• Easy to train
• Simply gain accuracy from depth
• Well transferrable

---

Abstract

深层的神经网络很难去训练，所以我们提出了一个残差学习框架去应对这个问题。所提出的网络更容易优化、能够通过显着增加的深度获得准确性。并且网络在ILSVRC 2015比赛上获得第一名的成绩，并且该网络具有很好的泛化性，可以应用到各种计算机视觉任务（检测、定位、分割）中。

---

Introduction

• 神经网络的深度很重要：深度网络自然地以端到端的多层方式集成低/中/高级特征和分类器，并且特征的“级别”可以通过堆叠层的数量（深度）来丰富。
• 直接堆叠层数的问题：梯度消失/梯度爆炸
  • 可以通过适当的权重初始化+Batch Normalization实现
• 退化现象：即使更深的网络开始收敛了，随着深度的增加，准确率也会趋于饱和，之后迅速下降。这个问题不是过拟合引起的，作者的例子证明层数越深，训练误差也会增加
• 构想出一个解决方案：模型不需要拟合底层的映射，只需要拟合相对于输入的残差（图更容易理解）。而且这种方案还不会增加额外的计算复杂度
  

---

Deep Residual Learning

网络结构很容易理解

三种投影方式：(A) 零填充方式用于增加维度，并且是无参数的； (B) 需要增加维度就使用投影，其他为恒等； © 所有shortcuts都是投影。

---

Experiments

利用不同的数据集验证模型泛化能力，都取得了不错的效果。

---

新学习的概念

泛函的距离空间（泛函分析）

泛函的最优逼近

网络架构的自动选取

---

本文心得

• ResNet解决网络退化的机理：

  • 深层梯度回传顺畅：防止梯度消失
  • 类比其他机器学习模型：
    • 集成学习boosting，每一个弱分类器拟合“前面的模型与GT之差”
    • 长短时记忆神经网络LSTM的遗忘门
    • ReLU激活函数（重要的时候让它输出，不重要的时候为0）

• 传统线性结构网络难以拟合“恒等映射”

  • skip connection可以让模型自行选择要不要更新
  • 弥补了高度非线性造成的不可逆的信息损失（MobileNet V2）

• 神经网络的可解释性分析：

  The Shattered Gradients Problem: If resnets are the answer, then what is the question?https://arxiv.org/abs/1702.08591

  Residual Networks Behave Like Ensembles of Relatively Shallow Networks：https://arxiv.org/abs/1605.06431

  为什么ResNet有效？

• BN层

  • 有BN不需要Bias

---

附录：代码实践

• 残差结构

import os.path

import torch.nn as nn
import torch

# 定义残差结构（18层和34层）
class BasicBlock(nn.Module):
    expansion = 1 # 一个残差块中卷积层的卷积核个数的变化
    def __init__(self, in_channel, out_channel, stride=1, downsample=None):
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=in_channel, out_channels=out_channel,
                               kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channel)
        self.relu = nn.ReLU()
        self.conv2 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel,
                               kernel_size=3, stride=1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channel)
        self.downsample = downsample

    def forward(self, x):
        identity = x
        if self.downsample is not None:
            identity = self.downsample(x)

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        out += identity
        out = self.relu(out)

        return out


# 定义残差结构（50层、101层、152层）
class Bottleneck(nn.Module):
    expansion = 4
    def __init__(self, in_channel, out_channel, stride=1, downsample=None):
        super(Bottleneck, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=in_channel, out_channels=out_channel,
                               kernel_size=1, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channel)

        self.conv2 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel,
                               kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channel)

        self.conv3 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel * self.expansion,
                               kernel_size=1, stride=1, padding=1, bias=False)
        self.bn3 = nn.BatchNorm2d(out_channel * self.expansion)

        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample

    def forward(self, x):
        identity = x
        #  不太明白为什么这样设计，当残差块上面identity和最后输出大小不一致时，不应该一定要用卷积操作吗，这里如果是None，不是大小就不合适了吗
        if self.downsample is not None:
            identity = self.downsample(x)
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)

        out += identity
        out =  self.relu(out)

        return out

• ResNet

class ResNet(nn.Module):
    def __init__(self, block, block_num, num_classes=1000, include_top=True):
        super(ResNet, self).__init__()
        self.include_top = include_top
        self.in_channel = 64
        self.conv1 = nn.Conv2d(3, out_channels=self.in_channel,
                               kernel_size=7, stride=2, padding=3, bias=False)
        self.bn1 = nn.BatchNorm2d(self.in_channel)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self.make_layer(block, 64, block_num[0])
        self.layer2 = self.make_layer(block, 128, block_num[1], stride=2)
        self.layer3 = self.make_layer(block, 256, block_num[2], stride=2)
        self.layer4 = self.make_layer(block, 512, block_num[3], stride=2)

        if self.include_top:
            self.avgpool = nn.AdaptiveAvgPool2d((1,1))
            self.fc = nn.Linear(512 * block.expansion, num_classes)

        # 对卷积层进行初始化
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')

    '''
        block:各种残差块
        channel:残差结构中卷积层1的卷积核个数
        block_num:包含了多少个残差结构
        stride
    '''
    def make_layer(self, block, channel, block_num, stride=1):
        downsample = None
        if stride != 1 or self.in_channel != channel * block.expansion:
            downsample = nn.Sequential(
                nn.Conv2d(self.in_channel, channel * block.expansion, kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(channel * block.expansion)
            )
        layers = []
        layers.append(block(self.in_channel, channel, downsample=downsample, stride=stride))
        self.in_channel = channel * block.expansion

        for _ in range(1, block_num):
            layers.append(block(self.in_channel, channel))
        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.maxpool(out)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)

        if self.include_top:
            out = self.avgpool(out)
            out = torch.flatten(out, 1)
            out = self.fc(out)
        return out


def resnet34(num_classes=1000, include_top=True):
    # https://download.pytorch.org/models/resnet34-333f7ec4.pth
    return ResNet(BasicBlock, [3, 4, 6, 3], num_classes=num_classes, include_top=include_top)

def resnet101(num_classes=1000, include_top=True):
    return ResNet(Bottleneck, [3, 4, 23, 3], num_classes=num_classes, include_top=include_top)

• 训练代码

import torchvision.models.resnet
from torchvision import transforms, datasets
import os
import json
import torch.optim as optim
from tqdm import tqdm
import sys

device = torch.device('cuda:0'if torch.cuda.is_available() else 'cpu')
print("using {} device".format(device))

data_transform = {
    "train": transforms.Compose([transforms.RandomResizedCrop(224),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.ToTensor(),
                                 transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]),
    "val": transforms.Compose([transforms.Resize(256),
                               transforms.CenterCrop(224),
                               transforms.ToTensor(),
                               transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])}


data_root = os.path.abspath(os.path.join(os.getcwd(), "../.."))
image_path = os.path.join(data_root, 'dataset', 'flower_photos')
assert os.path.exists(image_path), '{} path does not exist.'.format(image_path)
train_dataset = datasets.ImageFolder(root=os.path.join(image_path, "train"),
                                     transform=data_transform['train'])
train_num = len(train_dataset)
# {'daisy': 0, 'dandelion': 1, 'roses': 2, 'sunflowers': 3, 'tulips': 4}
flower_list = train_dataset.class_to_idx
# {0: 'daisy', 1: 'dandelion', 2: 'roses', 3: 'sunflowers', 4: 'tulips'}
cla_dict = dict((val, key) for key, val in flower_list.items())
json_str = json.dumps(cla_dict, indent=4) # indent=4 参数指定了缩进为 4 个空格
with open('class_indices.json', 'w') as json_file:
    json_file.write(json_str)


batch_size = 16
# nw = min([os.cpu_count(), batch_size if batch_size > 1 else 0, 8])
nw = 0
print('Using {} dataloader workers every process'.format(nw))

train_loader = torch.utils.data.DataLoader(train_dataset,
                                           batch_size=batch_size, shuffle=True,num_workers=nw)

validate_dataset = datasets.ImageFolder(root=os.path.join(image_path, 'val'),
                                        transform=data_transform['val'])
val_num = len(validate_dataset)
validate_loader = torch.utils.data.DataLoader(validate_dataset, batch_size=batch_size, shuffle=False, num_workers=nw)
print("using {} images for training, {} images for validation.".format(train_num, val_num))


net = resnet34()
model_weight_path = './resnet34-pre.pth'
assert os.path.exists(model_weight_path), "file {} does not exist.".format(model_weight_path)
# load pretrain weights
# 载入预训练模型参数的一种方法
net.load_state_dict(torch.load(model_weight_path, map_location='cpu'))
in_channel = net.fc.in_features # 输入特征矩阵的深度
net.fc = nn.Linear(in_channel, 5)
net.to(device)
# 另一种方法可以是可以先载入内存，之后再删掉fc

loss_function = nn.CrossEntropyLoss()

params = [p for p in net.parameters() if p.requires_grad]
optimizer = optim.Adam(params, lr=0.0001)

epochs = 3
best_acc = 0.0
save_path = './resnet34.pth'
train_steps = len(train_loader)


for epoch in range(epochs):
    # train
    net.train()
    running_loss = 0.0
    train_bar = tqdm(train_loader, file=sys.stdout)
    for step, data in enumerate(train_bar):
        images, labels = data
        optimizer.zero_grad()
        logits = net(images.to(device))
        loss = loss_function(logits, labels.to(device))
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        train_bar.desc = "train epoch[{}/{}] loss:{:.3f}".format(epoch + 1,
                                                                 epochs,
                                                                 loss)

    # validate
    net.eval()
    acc = 0.0  # accumulate accurate number / epoch
    with torch.no_grad():
        val_bar = tqdm(validate_loader, file=sys.stdout)
        for val_data in val_bar:
            val_images, val_labels = val_data
            outputs = net(val_images.to(device))
            # loss = loss_function(outputs, test_labels)
            predict_y = torch.max(outputs, dim=1)[1]
            acc += torch.eq(predict_y, val_labels.to(device)).sum().item()

            val_bar.desc = "valid epoch[{}/{}]".format(epoch + 1,
                                                       epochs)

    val_accurate = acc / val_num
    print('[epoch %d] train_loss: %.3f  val_accuracy: %.3f' %
          (epoch + 1, running_loss / train_steps, val_accurate))

    if val_accurate > best_acc:
        best_acc = val_accurate
        torch.save(net.state_dict(), save_path)

print('Finished Training')

• 推理代码

import os
import json

import torch
from PIL import Image
from torchvision import transforms
import matplotlib.pyplot as plt

from model import resnet34

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
data_transform = transforms.Compose(
    [transforms.Resize(256),
     transforms.CenterCrop(224),
     transforms.ToTensor(),
     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

# load image
img_path = r"D:\workplace\dl\cnn\sun_flower.jpg"
assert os.path.exists(img_path), "file: '{}' dose not exist.".format(img_path)
img = Image.open(img_path)
plt.imshow(img)
# [N, C, H, W]
img = data_transform(img)
# expand batch dimension
img = torch.unsqueeze(img, dim=0)

# read class_indict
json_path = './class_indices.json'
assert os.path.exists(json_path), "file: '{}' dose not exist.".format(json_path)

with open(json_path, "r") as f:
    class_indict = json.load(f)

# create model
model = resnet34(num_classes=5).to(device)

# load model weights
weights_path = "./resNet34.pth"
assert os.path.exists(weights_path), "file: '{}' dose not exist.".format(weights_path)
model.load_state_dict(torch.load(weights_path, map_location=device))

# prediction
model.eval()
with torch.no_grad():
    # predict class
    output = torch.squeeze(model(img.to(device))).cpu()
    predict = torch.softmax(output, dim=0)
    predict_cla = torch.argmax(predict).numpy()

print_res = "class: {}   prob: {:.3}".format(class_indict[str(predict_cla)],
                                             predict[predict_cla].numpy())
plt.title(print_res)
for i in range(len(predict)):
    print("class: {:10}   prob: {:.3}".format(class_indict[str(i)],
                                              predict[i].numpy()))
plt.show()

其他

微软亚洲研究院

学术追星