【OpenGait 之 GaitSet 系列】 了解 GaitSet 代码与原论文算法框架图的对应关系以及算法流程

最近在使用AutoDL云服务器跑步态识别的代码，用的是开放步态识别框架OpenGait，记录一下自己跑代码的步骤和遇到的问题以及解决方式。

这篇文章主要记录OpenGait项目里GaitSet代码与原论文框架图的对应关系以及算法流程。

---

一、GaitSet论文算法框架图

论文题目：《Gaitset: Regarding gait as a set for cross-view gait recognition》
下载地址：GaitSet2019

注：GaitSet最初版本是2019年发布的，2021年该团队又在2019年版本基础上发表了另一篇文章，题目为《GaitSet: Cross-view gait recognition through utilizing gait as a deep set》，21年和19年文章内容大致相同。2021版本下载地址为：GaitSet2021

Gaitset算法框架图如下：

二、OpenGait项目里的GaitSet代码

为什么要强调“OpenGait项目里的GaitSet代码”？因为源代码和OpenGait项目里的在组织结构上不太一样，本文只是针对OpenGait项目里的。

OpenGait项目里的GaitSet代码文件路径为：OpenGait/opengait/modeling/models/gaitset.py

gaitset.py内容如下：

import torch
import copy
import torch.nn as nn

from ..base_model import BaseModel
from ..modules import SeparateFCs, BasicConv2d, SetBlockWrapper, HorizontalPoolingPyramid, PackSequenceWrapper

class GaitSet(BaseModel):
    def build_network(self, model_cfg):
        in_c = model_cfg['in_channels']
        self.set_block1 = nn.Sequential(BasicConv2d(in_c[0], in_c[1], 5, 1, 2),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[1], in_c[1], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        nn.MaxPool2d(kernel_size=2, stride=2))

        self.set_block2 = nn.Sequential(BasicConv2d(in_c[1], in_c[2], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[2], in_c[2], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        nn.MaxPool2d(kernel_size=2, stride=2))

        self.set_block3 = nn.Sequential(BasicConv2d(in_c[2], in_c[3], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[3], in_c[3], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True))

        self.gl_block2 = copy.deepcopy(self.set_block2)
        self.gl_block3 = copy.deepcopy(self.set_block3)
        
        self.set_block1 = SetBlockWrapper(self.set_block1)
        self.set_block2 = SetBlockWrapper(self.set_block2)
        self.set_block3 = SetBlockWrapper(self.set_block3)
        
        self.Head = SeparateFCs(**model_cfg['SeparateFCs'])

        self.set_pooling = PackSequenceWrapper(torch.max)

        self.HPP = HorizontalPoolingPyramid(bin_num=model_cfg['bin_num'])
        
    def forward(self, inputs):
        ipts, labs, _, _, seqL = inputs
        sils = ipts[0]  # [n, s, h, w]
        if len(sils.size()) == 4:
            sils = sils.unsqueeze(1)

        del ipts
        
        outs = self.set_block1(sils)
        gl = self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = self.gl_block2(gl)

        outs = self.set_block2(outs)
        gl = gl + self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = self.gl_block3(gl)

        outs = self.set_block3(outs)
        outs = self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = gl + outs

        feature1 = self.HPP(outs)  # [n, c, p]
        feature2 = self.HPP(gl)  # [n, c, p]
        feature = torch.cat([feature1, feature2], -1)  # [n, c, p]
        embs = self.Head(feature)

        n, _, s, h, w = sils.size()
        retval = {
            'training_feat': {
                'triplet': {'embeddings': embs, 'labels': labs}
            },
            'visual_summary': {
                'image/sils': sils.view(n*s, 1, h, w)
            },
            'inference_feat': {
                'embeddings': embs
            }
        }
        return retval

三、GaitSet 代码与论文算法框架图的对应关系

1、3个set_block：提取每一帧的特征
①代码：

        self.set_block1 = nn.Sequential(BasicConv2d(in_c[0], in_c[1], 5, 1, 2),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[1], in_c[1], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        nn.MaxPool2d(kernel_size=2, stride=2))

        self.set_block2 = nn.Sequential(BasicConv2d(in_c[1], in_c[2], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[2], in_c[2], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        nn.MaxPool2d(kernel_size=2, stride=2))

        self.set_block3 = nn.Sequential(BasicConv2d(in_c[2], in_c[3], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True),
                                        BasicConv2d(in_c[3], in_c[3], 3, 1, 1),
                                        nn.LeakyReLU(inplace=True))

②对应原文算法图位置：

2、2个gl_block：提取所有帧集合后的特征
①代码

        self.gl_block2 = copy.deepcopy(self.set_block2)
        self.gl_block3 = copy.deepcopy(self.set_block3)

②对应原文算法图位置：

③注：gl_block只有2个，分别是gl_block2和gl_block3，没有gl_block1，我认为是为了和set_block的数字对应

3、SetBlockWrapper操作：不对应原文图片内容，只是为了把3个set_block 封装起来

        self.set_block1 = SetBlockWrapper(self.set_block1)
        self.set_block2 = SetBlockWrapper(self.set_block2)
        self.set_block3 = SetBlockWrapper(self.set_block3)

4、SeparateFCs：独立全连接操作
①代码

        self.Head = SeparateFCs(**model_cfg['SeparateFCs'])

②对应原文算法图位置：

5、set_pooling：集合池化操作
①代码：

       self.set_pooling = PackSequenceWrapper(torch.max)

②对应原文算法图位置：

6、HPP：水平金字塔池化操作
①代码：

       self.HPP = HorizontalPoolingPyramid(bin_num=model_cfg['bin_num'])

②对应原文算法图位置：

四、GaitSet 代码与算法流程对应关系

1、流程1
①代码

        outs = self.set_block1(sils)
        gl = self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = self.gl_block2(gl)

②流程对应关系：

③解释：sils经过set_block1输出为蓝色out，所有蓝色out经过set_pooling操作输出为绿色gl，绿色gl经过gl_block2输出为正方体绿色gl，接下来的流程参考该解释

2、流程2
①代码

        outs = self.set_block2(outs)
        gl = gl + self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = self.gl_block3(gl)

②流程对应关系：

3、流程3
①代码：

        outs = self.set_block3(outs)
        outs = self.set_pooling(outs, seqL, options={"dim": 2})[0]
        gl = gl + outs

②流程对应关系：

4、流程4
①代码

        feature1 = self.HPP(outs)  # [n, c, p]
        feature2 = self.HPP(gl)  # [n, c, p]
        feature = torch.cat([feature1, feature2], -1)  # [n, c, p]
        embs = self.Head(feature)

②流程对应关系：

---

总结

欢迎使用OpenGait研究步态识别的伙伴进行交流