【深度学习】无监督图像分类的一些尝试

提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档

---

前言

因为一些项目需求，本人最近对无监督图像分类有一些想法，做了许多尝试最后得出了两种方案

---

一、初次尝试

最开始是想通过数字图像处理的一些思路来获取图像特征，然后使用kmeans聚类来进行无监督图像分类，特征指标用了RBG直方图合并，轮廓等，然后进行kmeans聚类，最后的结果发现效果不是很行，因为对于一个物体来说出了颜色和轮廓特征还有很多深层的特征，这就需要使用CNN来进行特征提取。但是这个时候就出现了一个问题，这是一个无监督的分类问题，我没有标签数据来进行计算损失。所以就想到了两个方向：1.使用预训练的resnet50来做迁移学习
2.使用无监督学习上的经典网络-自编码器(autoencoder)

二.使用迁移学习来提取特征

使用resnet50的原因是因为：
1.resnet50有很深的网络层级，可以用来当特征提取器
2.resnet50是在大规模的数据集ImageNet上做的预训练，这种迁移学习的方法通常能够带来更好的性能，尤其是在有限的数据集上。

1.导入resnet50

代码如下（示例）：

import cv2
import numpy as np
import os
import warnings
import time
from tqdm import tqdm
import  shutil
import torch
import torch.nn as nn
from torchvision.models import resnet50
from  torchvision.transforms import transforms
model = resnet50(pretrained=True).cuda()

2.获取图片数据

def get_image_data(filepath:str) -> list:
    '''
        get images
    :param filepath:
        image files' path
    :return:
    image_data:
        a list of images
    '''
    image_data = []
    for root, dirs, files in os.walk(filepath):
        for file in files:
            if file.lower().endswith(('.png', '.jpg', '.jpeg')):
                image_data.append(cv2.cvtColor(cv2.imread(os.path.join(root, file)),cv2.COLOR_BGR2RGB))
    return image_data

3.处理图片数据

    def extract_features(image_data,model)->list:
        '''
            :param image_data:
        	a list of images
        	model:
        	neanmodel
            use resnet50 to extract the feature_map
        '''
        features = []
        transform = transforms.Compose([
            transforms.ToPILImage(),
            transforms.Resize((224, 224)),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ])
        for image in image_data:
            img = transform(image)
            img = img.unsqueeze(0)
            with torch.no_grad():
                feature = model(img)
            features.append(feature.squeeze().numpy())
        return features

三、使用自编码器

最基本的一个想法就是使用自编码器的encoder-decoder结构，encoder中使用若干层卷积和BatchNorm，docoder中使用若干层反卷积+BatchNorm。使用自编码器来重构图像训练整个网络反过来训练encoder结构，然后使用训练好的encoder器来充当特征提取器。

1.自编码器网络结构

代码如下：

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Conv2d(4, 16, kernel_size=3, stride=2, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )

        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(32, 16, kernel_size=3, stride=2, padding=1, output_padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.ConvTranspose2d(16, 4, kernel_size=3, stride=2, padding=1, output_padding=1),
            nn.BatchNorm2d(4),
            nn.Sigmoid(),
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

    def encode(self,x):
        x = self.encoder(x)
        return x

    def decode(self,x):
        x = self.decoder(x)
        return x

2.训练过程

代码如下：

num_epochs = 10
for epoch in range(num_epochs):
    # 训练
    start = time.time()
    for batch in train_loader:
        images = batch.cuda()

        # 前向传递
        outputs = autoencoder(images)

        # 计算损失
        loss = criterion(outputs, images)

        # 反向传播优化
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    # 验证集测试
    val_loss = 0.0
    with torch.no_grad():
        for data in val_loader:
            val_images = data.cuda()
            val_outputs = autoencoder(val_images)
            val_loss += criterion(val_outputs, val_images).item()

    val_loss /= len(val_loader)
    print(f'Epoch [{epoch + 1}/{num_epochs}], Training Loss: {loss.item():.4f}, Validation Loss: {val_loss:.4f},Time:{time.time()-start:.4f} seconds')

四.聚类

1.Kmeans-GPU

由于已经使用了神经网络来进行特征提取，那也就会想将Kmeans聚类过程放到GPU上，这里我在Github上找到一个Kmeans的GPU版本贴一下原链接，为了方便我自己使用我改成了单例类模式

Github：Kmeans-GPU

下面是整合的单例类代码：

class Kmeans_GPU():
    @staticmethod
    def cosine_distance(obs, centers):
        obs_norm = obs / obs.norm(dim=1, keepdim=True)
        centers_norm = centers / centers.norm(dim=1, keepdim=True)
        cos = torch.matmul(obs_norm, centers_norm.transpose(1, 0))
        return 1 - cos

    @staticmethod
    def l2_distance(obs, centers):
        dis = ((obs.unsqueeze(dim=1) - centers.unsqueeze(dim=0)) ** 2.0).sum(dim=-1).squeeze()
        return dis

    @staticmethod
    def _kmeans_batch(obs: torch.Tensor,
                      k: int,
                      distance_function,
                      batch_size=0,
                      thresh=1e-5,
                      norm_center=False):
        # k x D
        centers = obs[torch.randperm(obs.size(0))[:k]].clone()
        history_distances = [float('inf')]
        if batch_size == 0:
            batch_size = obs.shape[0]
        while True:
            # (N x D, k x D) -> N x k
            segs = torch.split(obs, batch_size)
            seg_center_dis = []
            seg_center_ids = []
            for seg in segs:
                distances = distance_function(seg, centers)
                center_dis, center_ids = distances.min(dim=1)
                seg_center_ids.append(center_ids)
                seg_center_dis.append(center_dis)

            obs_center_dis_mean = torch.cat(seg_center_dis).mean()
            obs_center_ids = torch.cat(seg_center_ids)
            history_distances.append(obs_center_dis_mean.item())
            diff = history_distances[-2] - history_distances[-1]
            if diff < thresh:
                if diff < 0:
                    warnings.warn("Distance diff < 0, distances: " + ", ".join(map(str, history_distances)))
                break
            for i in range(k):
                obs_id_in_cluster_i = obs_center_ids == i
                if obs_id_in_cluster_i.sum() == 0:
                    continue
                obs_in_cluster = obs.index_select(0, obs_id_in_cluster_i.nonzero().squeeze())
                c = obs_in_cluster.mean(dim=0)
                if norm_center:
                    c /= c.norm()
                centers[i] = c
        return centers, history_distances[-1]

    @staticmethod
    def kmeans(obs: torch.Tensor, k: int,
               distance_function=l2_distance,
               iter=20,
               batch_size=0,
               thresh=1e-5,
               norm_center=False):
        """
               Performs k-means on a set of observation vectors forming k clusters.

               Parameters
               ----------
               obs : torch.Tensor
                  Each row of the M by N array is an observation vector.

               k : int
                  The number of centroids to generate. A code is assigned to
                  each centroid, which is also the row index of the centroid
                  in the code_book matrix generated.

                  The initial k centroids are chosen by randomly selecting
                  observations from the observation matrix.

               distance_function : function, optional
                  The function to calculate distances between observations and centroids.
                  Default value: l2_distance

               iter : int, optional
                  The number of times to run k-means, returning the codebook
                  with the lowest distortion. This parameter does not represent the
                  number of iterations of the k-means algorithm.

               batch_size : int, optional
                  Batch size of observations to calculate distances, if your GPU memory can NOT handle all observations.
                  Default value is 0, which will send all observations into distance_function.

               thresh : float, optional
                  Terminates the k-means algorithm if the change in
                  distortion since the last k-means iteration is less than
                  or equal to thresh.

               norm_center : False, optional
                  Whether to normalize the centroids while updating every centroid.

               Returns
               -------
               best_centers : torch.Tensor
                  A k by N array of k centroids. The i'th centroid
                  codebook[i] is represented with the code i. The centroids
                  and codes generated represent the lowest distortion seen,
                  not necessarily the globally minimal distortion.

               best_distance : float
                  The mean distance between the observations passed and the best centroids generated.
               """
        best_distance = float("inf")
        best_centers = None
        for i in range(iter):
            if batch_size == 0:
                batch_size == obs.shape[0]
            centers, distance = Kmeans_GPU._kmeans_batch(obs, k,
                                                         norm_center=norm_center,
                                                         distance_function=distance_function,
                                                         batch_size=batch_size,
                                                         thresh=thresh)
            if distance < best_distance:
                best_centers = centers
                best_distance = distance
        return best_centers, best_distance

    @staticmethod
    def product_quantization(data, sub_vector_size, k, **kwargs):
        centers = []
        for i in range(0, data.shape[1], sub_vector_size):
            sub_data = data[:, i:i + sub_vector_size]
            sub_centers, _ = Kmeans_GPU.kmeans(sub_data, k=k, **kwargs)
            centers.append(sub_centers)
        return centers

    @staticmethod
    def data_to_pq(data, centers):
        assert (len(centers) > 0)
        assert (data.shape[1] == sum([cb.shape[1] for cb in centers]))

        m = len(centers)
        sub_size = centers[0].shape[1]
        ret = torch.zeros(data.shape[0], m,
                          dtype=torch.uint8,
                          device=data.device)
        for idx, sub_vec in enumerate(torch.split(data, sub_size, dim=1)):
            dis = Kmeans_GPU.l2_distance(sub_vec, centers[idx])
            ret[:, idx] = dis.argmin(dim=1).to(dtype=torch.uint8)
        return ret

    @staticmethod
    def train_product_quantization(data, sub_vector_size, k, **kwargs):
        center_list = Kmeans_GPU.product_quantization(data, sub_vector_size, k, **kwargs)
        pq_data = Kmeans_GPU.data_to_pq(data, center_list)
        return pq_data, center_list

    @staticmethod
    def _gram(x):
        (bs, ch, h, w) = x.size()
        f = x.view(bs, ch, w * h)
        f_T = f.transpose(1, 2)
        G = f.bmm(f_T) / (ch * h * w)
        return G

    @staticmethod
    def pq_distance_book(pq_centers):
        assert (len(pq_centers) > 0)

        pq = torch.zeros(len(pq_centers),
                         len(pq_centers[0]),
                         len(pq_centers[0]),
                         device=pq_centers[0].device)
        for ci, center in enumerate(pq_centers):
            for i in range(len(center)):
                dis = Kmeans_GPU.l2_distance(center[i:i + 1, :], center)
                pq[ci, i] = dis
        return pq

    @staticmethod
    def asymmetric_table(query, centers):
        m = len(centers)
        sub_size = centers[0].shape[1]
        ret = torch.zeros(
            query.shape[0], m, centers[0].shape[0],
            device=query.device)
        assert (query.shape[1] == sum([cb.shape[1] for cb in centers]))
        for i, offset in enumerate(range(0, query.shape[1], sub_size)):
            sub_query = query[:, offset: offset + sub_size]
            ret[:, i, :] = Kmeans_GPU.l2_distance(sub_query, centers[i])
        return ret

    @staticmethod
    def asymmetric_distance_slow(asymmetric_tab, pq_data):
        ret = torch.zeros(asymmetric_tab.shape[0], pq_data.shape[0])
        for i in range(asymmetric_tab.shape[0]):
            for j in range(pq_data.shape[0]):
                dis = 0
                for k in range(pq_data.shape[1]):
                    sub_dis = asymmetric_tab[i, k, pq_data[j, k].item()]
                    dis += sub_dis
                ret[i, j] = dis
        return ret

    @staticmethod
    def asymmetric_distance(asymmetric_tab, pq_data):
        pq_db = pq_data.long()
        dd = [torch.index_select(asymmetric_tab[:, i, :], 1, pq_db[:, i]) for i in range(pq_data.shape[1])]
        return sum(dd)

    @staticmethod
    def pq_distance(obj, centers, pq_disbook):
        ret = torch.zeros(obj.shape[0], centers.shape[0])
        for obj_idx, o in enumerate(obj):
            for ct_idx, c in enumerate(centers):
                for i, (oi, ci) in enumerate(zip(o, c)):
                    ret[obj_idx, ct_idx] += pq_disbook[i, oi.item(), ci.item()]
        return ret

2.聚类过程

使用了tqmd和shutil库来进行对图片直接分类
代码如下：

    def cluster_GPU(image_paths:list,features:list,k:int,savepath:str):
        '''
        :param image_paths:
            images'path;
        :param features:
            a list of images'feature map;
        :param k:
            cluster center nums;
        :param savepath:
            cluster's results' savepath
        :return:
        '''
        features_gpu = torch.from_numpy(np.array(features)).to('cuda')
        for i in range(k):
            cluster_folder = os.path.join(savepath, f'Cluster_{i}')
            os.makedirs(cluster_folder, exist_ok=True)
        centers, _ = Kmeans_GPU.kmeans(features_gpu.half(), k, distance_function=Kmeans_GPU.cosine_distance, iter=10)
        for i, image_path in tqdm(enumerate(image_paths), desc="Copying images"):
            distances = Kmeans_GPU.cosine_distance(features_gpu[i].unsqueeze(0).half(), centers)
            cluster_label = torch.argmin(distances).item()
            cluster_folder = os.path.join(savepath, f'Cluster_{cluster_label}')
            # 构造新的文件路径
            new_image_path = os.path.join(cluster_folder, os.path.basename(image_path))
            # 复制图片
            shutil.copy(image_path, new_image_path)

总结

本次项目尝试了两种不同的方式来进行无监督图像分类，主要的思想还是使用CNN来进行图像深度特征提取，然后使用kmeans进行聚类分析，对于要分成多少类，即k的个数可以使用网格搜索的思想去进行遍历寻找到最优k，这里也贴一份寻找最优k的代码块：
代码如下：

    def get_best_k(features,min:int,max:int):
        distortions = []  # 存储每个 K 对应的畸变值
        features_gpu = torch.from_numpy(np.array(features)).to('cuda')
        for n_clusters in range(min,max+1):
            print(f"Running K-means for K = {n_clusters}")
            # 使用GPU加速的K均值聚类
            centers, _ = Kmeans_GPU.kmeans(features_gpu.half(), n_clusters, distance_function=Kmeans_GPU.kmeans.cosine_distance, iter=10)

            # 将数据移到 CPU 上并转换为单精度
            centers = centers.float().cpu()
            features_gpu = features_gpu.float().cpu()

            # 计算距离
            distances = Kmeans_GPU.cosine_distance(features_gpu, centers)

            # 将数据移到 GPU 上
            distances = distances.to('cuda')

            # 计算畸变
            distortion = distances.min(dim=1).values.mean().item()
            distortions.append(distortion)
            features_gpu = features_gpu.to('cuda')

        # 找到畸变下降趋势平稳的 K 值
        best_k = min + np.argmin(np.gradient(distortions))  # 使用畸变梯度变化最小的 K 值

        print(f"The best K value is: {best_k}")
        return best_k