猫狗大战--经典图像分类题实战

最新推荐文章于 2024-06-02 07:18:16 发布

@op

最新推荐文章于 2024-06-02 07:18:16 发布

阅读量291

点赞数

文章标签：深度学习人工智能

本文链接：https://blog.csdn.net/qq_49327785/article/details/129770542

版权

一、使用EfficientNet实现图像分类（pytorch）

1.1 EfficientNet基本原理

1.2 EfficientNet实战

1.2.1 新建项目，安装并导入EfficientNet的库

1.2.2 设置全局参数

1.2.3 图像预处理、读取数据、导入数据

1.2.4 设置模型

1.2.5 定义训练过程和验证过程并验证

1.2.6 预测

一、使用EfficientNet实现图像分类（pytorch）

参考：https://zhuanlan.zhihu.com/p/498713239

AI研习社练习赛链接：https://god.yanxishe.com/41?from=god_home_list

因为本电脑只有核显没有独显，只能设置在cpu上进行训练。

1.1 EfficientNet基本原理

添加

1.2 EfficientNet实战

1.2.1 新建项目，安装并导入EfficientNet的库

1.2.2 设置全局参数

设置BatchSize、学习率和epochs，判断是否有cuda环境，如果没有设置为cpu。最好是将EfficientNet对应版本的pth下载到C盘users里面的文件夹中，会比在运行代码的过程中从网上下载更便捷一些。如C:\Users\wugua\.cache\torch\hub\checkpoints里.

下载预训练的网址在网上都能搜到，以efficientnet-b3-5fb5a3c3.pth为例，直接搜索即可，大部分是在github上下。

这是一位超级好人分享的网盘链接，包含常用的各种预训练模型的链接：

网络库网盘链接

import torch.optim as optim
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim
import torch.utils.data
import torch.utils.data.distributed
import torchvision.transforms as transforms
from dataset.dataset import DogCat
from torch.autograd import Variable
from efficientnet_pytorch import EfficientNet
# pip install efficientnet_pytorch

# 设置全局参数
modellr = 1e-4
BATCH_SIZE = 32
EPOCHS = 10
DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

1.2.3 图像预处理、读取数据、导入数据

train数据集的transform和验证集的transform分开做，train的图像处理出了resize和归一化之外，还可以设置图像的增强，比如旋转、随机擦除等一系列的操作，验证集则不需要做图像增强，另外不要盲目的做增强，不合理的增强手段很可能会带来负作用，甚至出现Loss不收敛的情况。

# 数据预处理

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])

])
transform_test = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])
dataset_train = DogCat(r'E:\A-DengNiran\ML\efficientnet实战\fenlei\data\train', transforms=transform, train=True)
dataset_test = DogCat(r"E:\A-DengNiran\ML\efficientnet实战\fenlei\data\test", transforms=transform_test, train=False)

# 读取数据

print(dataset_train.imgs)

# 导入数据

train_loader = torch.utils.data.DataLoader(dataset_train, batch_size=BATCH_SIZE, shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset_test, batch_size=BATCH_SIZE, shuffle=False)

DogCat()函数的调用，import了dataset.py文件，是对导入的data\train和data\test文件进行数据处理，代码如下：

# coding:utf8
import os
from PIL import Image
from torch.utils import data
from torchvision import transforms as T
from sklearn.model_selection import train_test_split
import glob

class DogCat(data.Dataset):

    def __init__(self, root, transforms=None, train=True, test=False):
        """
        主要目标： 获取所有图片的地址，并根据训练，验证，测试划分数据
        """
        self.test = test
        self.transforms = transforms
        imgs = [os.path.join(root, img) for img in os.listdir(root)]
        if self.test:  # 如果是test数据集
            imgs = sorted(imgs, key=lambda x: int(x.split('.')[-2].split('\\')[-1]))
        else:   # 如果是train数据集
            imgs = sorted(imgs, key=lambda x: int(x.split('.')[-2]))

        if self.test:  # 如果是test数据集
            self.imgs = imgs
        else:   # 如果是train数据集，还要再拆分为训练集和测试集
            trainval_files, val_files = train_test_split(imgs, test_size=0.3, random_state=42)
            if train: 
                self.imgs = trainval_files
            else:
                self.imgs = val_files

    def __getitem__(self, index):
        """
        一次返回一张图片的数据
        """
        img_path = self.imgs[index]
        if self.test:
            label =-1
        else:
            label = 1 if 'dog' in img_path.split('/')[-1] else 0
        data = Image.open(img_path)
        data = self.transforms(data)
        return data, label

    def __len__(self):
        return len(self.imgs)

1.2.4 设置模型

使用CrossEntropyLoss作为loss，模型采用efficientnet-B3。更改最后一层的全连接，将类别设置为2，然后将模型放到DEVICE。优化器选用Adam。

# 实例化模型并且移动到GPU
criterion = nn.CrossEntropyLoss()
model_ft = EfficientNet.from_pretrained('efficientnet-b3')
num_ftrs = model_ft._fc.in_features
model_ft._fc = nn.Linear(num_ftrs, 2)
model_ft.to(DEVICE)
# 选择简单暴力的Adam优化器，学习率调低
optimizer = optim.Adam(model_ft.parameters(), lr=modellr)


def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    modellrnew = modellr * (0.1 ** (epoch // 50))
    print("lr:", modellrnew)
    for param_group in optimizer.param_groups:
        param_group['lr'] = modellrnew

1.2.5 定义训练过程和验证过程并验证

# 定义训练过程

def train(model, device, train_loader, optimizer, epoch):
    model.train()
    sum_loss = 0
    total_num = len(train_loader.dataset)
    print(total_num, len(train_loader))
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data).to(device), Variable(target).to(device)
        output = model(data)
        loss = criterion(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print_loss = loss.data.item()
        sum_loss += print_loss
        if (batch_idx + 1) % 50 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, (batch_idx + 1) * len(data), len(train_loader.dataset),
                       100. * (batch_idx + 1) / len(train_loader), loss.item()))
    ave_loss = sum_loss / len(train_loader)
    print('epoch:{},loss:{}'.format(epoch, ave_loss))


# 验证过程
def val(model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    total_num = len(test_loader.dataset)
    print(total_num, len(test_loader))
    with torch.no_grad():
        for data, target in test_loader:
            data, target = Variable(data).to(device), Variable(target).to(device)
            output = model(data)
            loss = criterion(output, target)
            _, pred = torch.max(output.data, 1)
            correct += torch.sum(pred == target)
            print_loss = loss.data.item()
            test_loss += print_loss
        correct = correct.data.item()
        acc = correct / total_num
        avgloss = test_loss / len(test_loader)
        print('\nVal set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            avgloss, correct, len(test_loader.dataset), 100 * acc))


# 训练

for epoch in range(1, EPOCHS + 1):
    adjust_learning_rate(optimizer, epoch)
    train(model_ft, DEVICE, train_loader, optimizer, epoch)
    val(model_ft, DEVICE, test_loader)
torch.save(model_ft, 'model.pth')

输出结果：

1.2.6 预测

放置在Test.py文件中

import torch.utils.data.distributed
import torchvision.transforms as transforms
import torchvision.datasets as datasets
from PIL import Image
from torch.autograd import Variable
import os
classes = ('cat', 'dog')
transform_test = transforms.Compose([
         transforms.Resize((224, 224)),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])

DEVICE = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = torch.load("model.pth", map_location='cpu')
model.eval()
model.to(DEVICE)

path='data/test/'
testList=os.listdir(path)
for file in testList:
        img=Image.open(path+file)
        img=transform_test(img)
        img.unsqueeze_(0) # 这是应该是因为网络的接收输入是一个mini-batch，image unsqueeze后第一个维度是留给batch size的
        img = Variable(img).to(DEVICE)  # variable是一种可以不断变化的变量，符合反向传播，参数更新的属性
        out=model(img)
        # Predict
        _, pred = torch.max(out.data, 1)
        print('Image Name:{},predict:{}'.format(file,classes[pred.data.item()]))

输出结果：