【Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition】

于 2022-10-13 17:23:21 发布
阅读量142 收藏
点赞数 1
文章标签： 深度学习 人工智能 机器学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/afadgas/article/details/127303950
版权

摘要部分：开头一句话指出motion representation在人体动作识别中起着至关重要的作用，然后就直接切入自己的模型的简介，本文引入了一个新的紧凑的motion representation，叫做optical flow guided feature（OFF）。OFF是从optical flow的定义中发源而来的，并且和optical flow正交。之后又介绍了一些其他OFF的特征，最后简介了一下实验情况。

既然是叫optical flow guided feature，又和optical flow正交，那么就要先介绍一下optical flow。传统的optical flow当中有一个很著名的限制，叫做brightness constant constraint

在这里插入图片描述
这里面I表示像素值，x,y,t分别是空间两个维度和时间一个维度，△x,y,t表示的是很小的变化量。而对于feature level，也就是从原始像素值中用函数得到的feature，也有类似的表述
在这里插入图片描述
也就是不仅亮度在时空上具有连续性，对于坐标值的小改变是不变的，由亮度值计算出来的feature也是具有这种稳定性的。用p=(x,y,t)来表示一个位置，那么将上式可改写为
在这里插入图片描述
如果两边都除以△t，那么就得到
在这里插入图片描述
这里面的vx和vy两个东西物理意义暂不明确，被称作volecity of feature point at p，feature点的速度，而对于x,y,t求的偏导数也就是对空间和时间求的梯度。

网络结构的概览可以用下图表述
在这里插入图片描述
整个网络有三个子网络，，三个子网络各有各的用处。分别是feature generation sub-network，OFF sub-network，classification sub-network。feature generation sub-network就用普通的CNN网络抓取出一些基础的feature，然后OFF sub-network再从中抓取出OFF feature，接着，这些feature会输入到一些堆叠在一起的residual block之中进一步获得更加精细的feature，这两个sub-network输出的feature再输入到classification sub-network得到最终的分类结果，更加精细的网络结构图可以见下图
在这里插入图片描述
首先，基础的feature f(I)是用加了ReLU和max-pooling的卷积网络抓取出来的，网络结构选择的是BN-Inception，这个feature generation network也可以用其他网络结构代替。OFF sub-network包含多个OFF unit，不同的unit使用f(I)不同深度的特征，就像图中所示。每一个OFF unit包含一个OFF layer，用来生成OFF，生成OFF的过程其实可以用下图表示
在这里插入图片描述
首先输入的feature map的每一个位置都会使用一个1×1的卷积核将feature的channel数变成所需的固定大小，本文中是无论输入多少都变成128维的。之后用sobel算子计算空间维度上的梯度，用相邻帧对应位置像素值相减获得时间维度的梯度，这些梯度都计算完了就是得到了OFF了，将他们连接在一起并且连接上上一个level输出的lower level OFF feature，之后输入到residual network中，sobel算子其实很简单，如下所示
在这里插入图片描述
这个是分别生成x和y方向的梯度数据，也就是一个固定权重的kernel，用来计算像素值的差值。

在连接不同OFF unit的residual block中，OFF的dimension还会进一步减小，节约计算资源。residual block使用的是ResNet-20，不使用batchnorm，作者声称是为了避免过拟合。此外，OFF unit其实是可以加到一般的CNN layer中来辅助模型的。

最后是classification sub-network，classification sub-network将不同来源（指的应该是figure3中的三个score，但是有一点很疑惑，就是t和t+△t不都是原video吗？这两个的输出score有啥区别呢）feature拿来，分别使用inner-product classifier得到相应的classification score，对于所有sampled frames得到的classification score（这里说的又像是从video中提出的每一帧feature map都要计算一个相应的classification score）通过取平均值的方式合并在一起。这里其实不是很清楚最终的分类结果到底是怎么输出的，之后作者又介绍了，采取和TSN一样的设置，video中不是每一个frame都参与计算的，是要抽样的，抽取出来的每一个frame对应一个segment（一般来讲segment不一定是frame），每一个segment都会输出一个class score，对于OFF sub-network的各个segment的输出score通过average pooling来得到一个sub-network level的score，为了获得video-level的score，还需要考虑feature generation sub-network的输出score，也可以采取同样的average pooling的方法进行处理。此外，本文的feature generation sub-network和OFF sub-network是分开训练的，第一阶段是用已有的手段训练feature generation sub-network，第二阶段是固定feature generation sub-network，然后训练OFF sub-network。

BNInception_OFF的模型结构如下所示：

from RGB_OFF import BNInception_OFF
import torch
import torch.nn as nn
from thop import profile


model = BNInception_OFF()

print(model)

input = torch.randn(1, 3, 224, 224)
flops, params = profile(model, inputs=(input, ))



#print(flops)
#print(params)

输出如下

C:\Users\24093\anaconda3\envs\tuduipytorch\python.exe D:/Optical-Flow-Guided-Feature-Pytorch/my_test.py
BNInception_OFF(
  (consensus): ConsensusModule()
  (conv1_7x7_s2): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
  (conv1_7x7_s2_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (conv1_relu_7x7): ReLU(inplace=True)
  (pool1_3x3_s2): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=True)
  (conv2_3x3_reduce): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))
  (conv2_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (conv2_relu_3x3_reduce): ReLU(inplace=True)
  (conv2_3x3): Conv2d(64, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv2_3x3_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (conv2_relu_3x3): ReLU(inplace=True)
  (pool2_3x3_s2): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=True)
  (inception_3a_1x1): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3a_1x1_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_1x1): ReLU(inplace=True)
  (inception_3a_3x3_reduce): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3a_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_3x3_reduce): ReLU(inplace=True)
  (inception_3a_3x3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3a_3x3_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_3x3): ReLU(inplace=True)
  (inception_3a_double_3x3_reduce): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3a_double_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_3a_double_3x3_1): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3a_double_3x3_1_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_double_3x3_1): ReLU(inplace=True)
  (inception_3a_double_3x3_2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3a_double_3x3_2_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_double_3x3_2): ReLU(inplace=True)
  (inception_3a_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_3a_pool_proj): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1))
  (inception_3a_pool_proj_bn): BatchNorm2d(32, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3a_relu_pool_proj): ReLU(inplace=True)
  (inception_3b_1x1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3b_1x1_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_1x1): ReLU(inplace=True)
  (inception_3b_3x3_reduce): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3b_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_3x3_reduce): ReLU(inplace=True)
  (inception_3b_3x3): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3b_3x3_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_3x3): ReLU(inplace=True)
  (inception_3b_double_3x3_reduce): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3b_double_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_3b_double_3x3_1): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3b_double_3x3_1_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_double_3x3_1): ReLU(inplace=True)
  (inception_3b_double_3x3_2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3b_double_3x3_2_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_double_3x3_2): ReLU(inplace=True)
  (inception_3b_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_3b_pool_proj): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3b_pool_proj_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3b_relu_pool_proj): ReLU(inplace=True)
  (inception_3c_3x3_reduce): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_3c_3x3_reduce_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3c_relu_3x3_reduce): ReLU(inplace=True)
  (inception_3c_3x3): Conv2d(128, 160, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
  (inception_3c_3x3_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3c_relu_3x3): ReLU(inplace=True)
  (inception_3c_double_3x3_reduce): Conv2d(320, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_3c_double_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3c_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_3c_double_3x3_1): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_3c_double_3x3_1_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3c_relu_double_3x3_1): ReLU(inplace=True)
  (inception_3c_double_3x3_2): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
  (inception_3c_double_3x3_2_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_3c_relu_double_3x3_2): ReLU(inplace=True)
  (inception_3c_pool): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=True)
  (inception_4a_1x1): Conv2d(576, 224, kernel_size=(1, 1), stride=(1, 1))
  (inception_4a_1x1_bn): BatchNorm2d(224, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_1x1): ReLU(inplace=True)
  (inception_4a_3x3_reduce): Conv2d(576, 64, kernel_size=(1, 1), stride=(1, 1))
  (inception_4a_3x3_reduce_bn): BatchNorm2d(64, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_3x3_reduce): ReLU(inplace=True)
  (inception_4a_3x3): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4a_3x3_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_3x3): ReLU(inplace=True)
  (inception_4a_double_3x3_reduce): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1))
  (inception_4a_double_3x3_reduce_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_4a_double_3x3_1): Conv2d(96, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4a_double_3x3_1_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_double_3x3_1): ReLU(inplace=True)
  (inception_4a_double_3x3_2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4a_double_3x3_2_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_double_3x3_2): ReLU(inplace=True)
  (inception_4a_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_4a_pool_proj): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4a_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4a_relu_pool_proj): ReLU(inplace=True)
  (inception_4b_1x1): Conv2d(576, 192, kernel_size=(1, 1), stride=(1, 1))
  (inception_4b_1x1_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_1x1): ReLU(inplace=True)
  (inception_4b_3x3_reduce): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1))
  (inception_4b_3x3_reduce_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_3x3_reduce): ReLU(inplace=True)
  (inception_4b_3x3): Conv2d(96, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4b_3x3_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_3x3): ReLU(inplace=True)
  (inception_4b_double_3x3_reduce): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1))
  (inception_4b_double_3x3_reduce_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_4b_double_3x3_1): Conv2d(96, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4b_double_3x3_1_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_double_3x3_1): ReLU(inplace=True)
  (inception_4b_double_3x3_2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4b_double_3x3_2_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_double_3x3_2): ReLU(inplace=True)
  (inception_4b_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_4b_pool_proj): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4b_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4b_relu_pool_proj): ReLU(inplace=True)
  (inception_4c_1x1): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1))
  (inception_4c_1x1_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_1x1): ReLU(inplace=True)
  (inception_4c_3x3_reduce): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4c_3x3_reduce_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_3x3_reduce): ReLU(inplace=True)
  (inception_4c_3x3): Conv2d(128, 160, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4c_3x3_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_3x3): ReLU(inplace=True)
  (inception_4c_double_3x3_reduce): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4c_double_3x3_reduce_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_4c_double_3x3_1): Conv2d(128, 160, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4c_double_3x3_1_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_double_3x3_1): ReLU(inplace=True)
  (inception_4c_double_3x3_2): Conv2d(160, 160, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4c_double_3x3_2_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_double_3x3_2): ReLU(inplace=True)
  (inception_4c_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_4c_pool_proj): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4c_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4c_relu_pool_proj): ReLU(inplace=True)
  (inception_4d_1x1): Conv2d(608, 96, kernel_size=(1, 1), stride=(1, 1))
  (inception_4d_1x1_bn): BatchNorm2d(96, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_1x1): ReLU(inplace=True)
  (inception_4d_3x3_reduce): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4d_3x3_reduce_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_3x3_reduce): ReLU(inplace=True)
  (inception_4d_3x3): Conv2d(128, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4d_3x3_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_3x3): ReLU(inplace=True)
  (inception_4d_double_3x3_reduce): Conv2d(608, 160, kernel_size=(1, 1), stride=(1, 1))
  (inception_4d_double_3x3_reduce_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_4d_double_3x3_1): Conv2d(160, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4d_double_3x3_1_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_double_3x3_1): ReLU(inplace=True)
  (inception_4d_double_3x3_2): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4d_double_3x3_2_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_double_3x3_2): ReLU(inplace=True)
  (inception_4d_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_4d_pool_proj): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4d_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4d_relu_pool_proj): ReLU(inplace=True)
  (inception_4e_3x3_reduce): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_4e_3x3_reduce_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4e_relu_3x3_reduce): ReLU(inplace=True)
  (inception_4e_3x3): Conv2d(128, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
  (inception_4e_3x3_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4e_relu_3x3): ReLU(inplace=True)
  (inception_4e_double_3x3_reduce): Conv2d(608, 192, kernel_size=(1, 1), stride=(1, 1))
  (inception_4e_double_3x3_reduce_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4e_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_4e_double_3x3_1): Conv2d(192, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_4e_double_3x3_1_bn): BatchNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4e_relu_double_3x3_1): ReLU(inplace=True)
  (inception_4e_double_3x3_2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
  (inception_4e_double_3x3_2_bn): BatchNorm2d(256, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_4e_relu_double_3x3_2): ReLU(inplace=True)
  (inception_4e_pool): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=True)
  (inception_5a_1x1): Conv2d(1056, 352, kernel_size=(1, 1), stride=(1, 1))
  (inception_5a_1x1_bn): BatchNorm2d(352, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_1x1): ReLU(inplace=True)
  (inception_5a_3x3_reduce): Conv2d(1056, 192, kernel_size=(1, 1), stride=(1, 1))
  (inception_5a_3x3_reduce_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_3x3_reduce): ReLU(inplace=True)
  (inception_5a_3x3): Conv2d(192, 320, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5a_3x3_bn): BatchNorm2d(320, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_3x3): ReLU(inplace=True)
  (inception_5a_double_3x3_reduce): Conv2d(1056, 160, kernel_size=(1, 1), stride=(1, 1))
  (inception_5a_double_3x3_reduce_bn): BatchNorm2d(160, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_5a_double_3x3_1): Conv2d(160, 224, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5a_double_3x3_1_bn): BatchNorm2d(224, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_double_3x3_1): ReLU(inplace=True)
  (inception_5a_double_3x3_2): Conv2d(224, 224, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5a_double_3x3_2_bn): BatchNorm2d(224, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_double_3x3_2): ReLU(inplace=True)
  (inception_5a_pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
  (inception_5a_pool_proj): Conv2d(1056, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_5a_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5a_relu_pool_proj): ReLU(inplace=True)
  (inception_5b_1x1): Conv2d(1024, 352, kernel_size=(1, 1), stride=(1, 1))
  (inception_5b_1x1_bn): BatchNorm2d(352, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_1x1): ReLU(inplace=True)
  (inception_5b_3x3_reduce): Conv2d(1024, 192, kernel_size=(1, 1), stride=(1, 1))
  (inception_5b_3x3_reduce_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_3x3_reduce): ReLU(inplace=True)
  (inception_5b_3x3): Conv2d(192, 320, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5b_3x3_bn): BatchNorm2d(320, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_3x3): ReLU(inplace=True)
  (inception_5b_double_3x3_reduce): Conv2d(1024, 192, kernel_size=(1, 1), stride=(1, 1))
  (inception_5b_double_3x3_reduce_bn): BatchNorm2d(192, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_double_3x3_reduce): ReLU(inplace=True)
  (inception_5b_double_3x3_1): Conv2d(192, 224, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5b_double_3x3_1_bn): BatchNorm2d(224, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_double_3x3_1): ReLU(inplace=True)
  (inception_5b_double_3x3_2): Conv2d(224, 224, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (inception_5b_double_3x3_2_bn): BatchNorm2d(224, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_double_3x3_2): ReLU(inplace=True)
  (inception_5b_pool): MaxPool2d(kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), dilation=(1, 1), ceil_mode=True)
  (inception_5b_pool_proj): Conv2d(1024, 128, kernel_size=(1, 1), stride=(1, 1))
  (inception_5b_pool_proj_bn): BatchNorm2d(128, eps=1e-05, momentum=0.9, affine=True, track_running_stats=True)
  (inception_5b_relu_pool_proj): ReLU(inplace=True)
  (global_pool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (last_linear): Linear(in_features=1024, out_features=1000, bias=True)
  (motion_conv_gen_3a): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_3a): Conv2d(256, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_3a): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_3b): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_3b): Conv2d(320, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_3b): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_3c): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_3c): Conv2d(576, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_3c): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_trans_28): Conv2d(320, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))
  (motion_conv1_trans_28a): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans_28a): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans_28a): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv_branch_28a): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv1_trans_28b): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans_28b): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans_28b): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv1_trans_28c): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans_28c): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans_28c): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv_gen_4a): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_4a): Conv2d(576, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_4a): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_4b): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_4b): Conv2d(576, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_4b): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_4c): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_4c): Conv2d(608, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_4c): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_4d): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_4d): Conv2d(608, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_4d): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_trans_14): Conv2d(1056, 128, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2))
  (motion_conv1_trans_14a): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans_14a): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans_14a): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv_expand_trans_14a): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv1_trans_14b): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans_14b): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans_14b): Conv2d(128, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv_gen_5a): Conv2d(1024, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_5a): Conv2d(1024, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_5a): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_gen_5b): Conv2d(1024, 128, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_down_5b): Conv2d(1024, 32, kernel_size=(1, 1), stride=(1, 1))
  (motion_spatial_grad_5b): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32)
  (motion_conv_trans): Conv2d(832, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv1_trans): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv2_trans): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (motion_conv3_trans): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
  (motion_conv_branch_trans): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1))
  (fc_action_motion): Linear(in_features=1024, out_features=101, bias=True)
  (fc_action_motion_28): Linear(in_features=256, out_features=101, bias=True)
  (fc_action_motion_14): Linear(in_features=512, out_features=101, bias=True)
  (motion_relu_gen_3a): ReLU()
  (motion_relu_gen_3b): ReLU()
  (motion_relu_gen_3c): ReLU()
  (motion_relu_trans_28a): ReLU()
  (motion_relu_trans_28b): ReLU()
  (motion_relu_trans_28c): ReLU()
  (motion_relu_gen_4a): ReLU()
  (motion_relu_gen_4b): ReLU()
  (motion_relu_gen_4c): ReLU()
  (motion_relu_gen_4d): ReLU()
  (motion_relu_trans_14): ReLU()
  (motion_relu_gen_5a): ReLU()
  (motion_relu_gen_5b): ReLU()
  (motion_relu_trans_7): ReLU()
  (motion_pool_trans_28): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=0, dilation=(1, 1), ceil_mode=True)
  (dropout): Dropout(p=0.8, inplace=False)
  (softmax): Softmax(dim=None)
)
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv2d'>.
[INFO] Register count_normalization() for <class 'torch.nn.modules.batchnorm.BatchNorm2d'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.activation.ReLU'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.pooling.MaxPool2d'>.
[INFO] Register count_avgpool() for <class 'torch.nn.modules.pooling.AvgPool2d'>.
[INFO] Register count_linear() for <class 'torch.nn.modules.linear.Linear'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.dropout.Dropout'>.
[INFO] Register count_softmax() for <class 'torch.nn.modules.activation.Softmax'>.

Process finished with exit code 0

计算BNInception_OFF的flops和params如下所示：

from RGB_OFF import BNInception_OFF
import torch
import torch.nn as nn
from thop import profile


model = BNInception_OFF()

#print(model)

input = torch.randn(1, 3, 224, 224)
flops, params = profile(model, inputs=(input, ))



print(flops)
print(params)

输出如下所示：

C:\Users\24093\anaconda3\envs\tuduipytorch\python.exe D:/Optical-Flow-Guided-Feature-Pytorch/my_test.py
[INFO] Register count_convNd() for <class 'torch.nn.modules.conv.Conv2d'>.
[INFO] Register count_normalization() for <class 'torch.nn.modules.batchnorm.BatchNorm2d'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.activation.ReLU'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.pooling.MaxPool2d'>.
[INFO] Register count_avgpool() for <class 'torch.nn.modules.pooling.AvgPool2d'>.
[INFO] Register count_linear() for <class 'torch.nn.modules.linear.Linear'>.
[INFO] Register zero_ops() for <class 'torch.nn.modules.dropout.Dropout'>.
[INFO] Register count_softmax() for <class 'torch.nn.modules.activation.Softmax'>.
2048408096.0
11295240.0

Process finished with exit code 0
Shiina丶Mashiro
关注
【论文笔记】视频分类系列 Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video (OFF)
elaine_bao的专栏
06-24 3266
Paper: Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (CVPR2018, SenseTime) 0. 简述 运动信息对于视频中动作的识别有着重要的作用。其中最常用的一种运动信息，是光流。之前的方法计算光流是offline的，比较耗时，...
Optical Flow Guided Feature 笔记
qq_40133804的博客
02-19 367
论文：《Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition》 这篇论文主要贡献点有两个：1、提出了一种鲁棒性强速度快的运动信息表示特征OFF。2、提出了将OFF运用于端到端训练的网络结构。 OFF 简单的说，OFF即是光流向量的正交向量。由基本的光流约束方...
读书笔记22：Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recogni
b224618的博客
09-09 1855
文章题目：Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition（CVPR2018） http://openaccess.thecvf.com/content_cvpr_2018/papers/Sun_Optical_Flow_Guided_CVPR_201...
Optical Flow Guided Feature A Fast and Robust Motion Representation for Video Action Recognition论文解读
(翁仲銘)Chung-MingOwn
11-21 417
Optical Flow Guided Feature A Fast and Robust Motion Representation for Video Action RecognitionOptical Flow Guided Feature A Fast and Robust Motion Representation for Video Action Recognition1. Abstr...
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
ssxx1a的博客
02-29 467
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition Abstract 1.提出了一个Optical Flow guided Feature(OFF)模块，可以用于video action recognition。直接理解，该模块可以直接计算deep fea...
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition 翻译
黄笳倞的博客
12-24 5940
光流引导特征：视频动作识别的快速鲁棒运动表示 项目地址：https://github.com/kevin-ssy/Optical-Flow-Guided-Feature 摘要 运动表示在视频中的人类动作识别中起着至关重要的作用。 在本研究中，我们介绍了一种用于视频动作识别的新颖紧凑运动表示，称为光流引导特征（OFF...
基于深度学习的Action Recognition（行为识别）【二】
热门推荐
事了拂衣去，深藏功与名
01-20 3万+
本文发表在：知乎专栏 看了近一个月的论文，对行为识别领域中目前主流的基于深度学习的方法算是有了基本的认识。但今天只做总结，关于每一篇论文的详细理解，之后有时间补上。也欢迎指正交流。 下图为目前主流模型的比较。其中T3D标称效果好于I3D，但由于结果是作者复现得来，故在这里不做比较。顺序自上向下按UCF101的准确率排列。
（十：2020.08.28）CVPR 2018 追踪之论文纲要（译）
Jojo论文基地
08-28 7250
CVPR 2018 追踪之论文纲要（修正于2020.08.28）讲在前面论文目录 讲在前面 论坛很多博客都对论文做了总结和分类，但就医学领域而言，对这些论文的筛选信息显然需要更加精细的把控，所以自己对这979篇的论文做一个大致从名称上的筛选，希望能找到些能解决当前问题的答案。 论文链接建议直接Google论文名，比去各种论文或顶会网站找不知道快捷多少。 下面皆为机器翻译以方便我第一次筛选，我会慢慢修正，但现在请结合。有兴趣的可以问我要处理这些论文并自动翻译的脚本。 Respect！ 论文目录
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition论文
Dongjiuqing的博客
02-24 600
这是我转载的别人的一片写的还不错的论文笔记，由于内容比较多，就没有自己写。这是一片2018年的CVPR上的一片文章，提出了一种基于光流引导的特征（OFF），将这个特征融入到CNN网络中 论文地址：http://openaccess.thecvf.com/content_cvpr_2018/html/Sun_Optical_Flow_Guided_CVPR_2018_paper.html Opt...
Motion and Optical Flow
kaggle expert，全球排名前1000，清华计算机研究生，兴趣算法工程
08-01 429
光流Optical Flow 定义：光流是图像中亮度模式的明显运动• 注意：明显的运动可能是由照明变化引起的，没有任何实际运动–考虑固定照明下的均匀旋转球体vs.运动照明下的静止球体 ...
【论文笔记】视频物体检测(VID)系列 NoScope:1000x的视频检索加速算法
elaine_bao的专栏
12-17 1732
计算机视觉的进展，特别是近期深度神经网络的进展，使得在不断增长的视频数据中进行检索成为可能。但是，基于神经网络去做大规模视频检索，直接的问题就是计算量激增。比如现在最快的物体检测器能达到1～2.5x real time的效率（30～80fps），前提是买一个$4000的GPU。
视频行为识别[1]Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
DavidZh
07-27 938
　2018年论文 ［１］Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition［CVPR2018］ 题目：光流引导特征：快速，稳健的视频运动表示（OFF） 摘要：运动表示在视频人体行为识别中扮演者重要角色.在本研究中,我们介绍一种新的紧凑的运动表征方法用于视频行为识别,...
【论文笔记】光流在视频行为识别中的作用
elaine_bao的专栏
07-02 1万+
Paper：On the Integration of Optical Flow and Action Recognition Author：Facebook，Zhejiang University，MPI for Intelligent Systems，NVIDIA 0. 引言 在做视频行为识别时，特别是基于two-stream框架时，常常会引入光流图作为双流网络其中一支的输入。这是很...
行为识别Action Detection概述及资源合集（持续更新...）
Gavinmiaoc的博客
07-24 2万+
随着深度学习技术的发展，以及计算能力的进步（GPU等），现在基于视频的研究领域越来越受到重视。视频与图片最大的不同在于视频还包含了时序上的信息，此外需要的计算量通常也大很多。 这篇主要介绍Action Recognition（行为识别）这个方向。这个方向的主要目标是判断一段视频中人的行为的类别，所以也可以叫做Human Action Recognition。虽然这个问题是针对视频中人的动作，但基...
CVPR 2018 | TVNet：可端到端学习视频的运动表征
沈春旭的博客
05-09 2287
“来自腾讯 AI Lab、MIT、清华、斯坦福大学的研究者完成并入选 CVPR 2018 Spotlight 论文的一项研究提出了一种能从数据中学习出类光流特征并且能进行端到端训练的神经网络：TVNet”尽管端到端的特征学习已经取得了重要的进展，但是人工设计的光流特征仍然被广泛用于各类视频分析任务中。为了弥补这个不足，由来自腾讯 AI Lab、MIT、清华、斯坦福大学的研究者完成并入选 CVPR ...
【论文笔记】视频分类系列 Appearance-and-Relation Networks for Video Classification (ARTNet)
elaine_bao的专栏
03-05 4949
时间空间域的特征学习对于视频分类来说很重要。文章提出了Appearance-and-Relation Networks（ARTNet）的结构，用于end-to-end地学习视频时间空间域的特征。 ARTNet是由多个名为SMART的block堆叠而成。SMART模块的目标是从RGB帧中分别学习到appearance和relation。
《DIP: Deep Inverse Patchmatch for High-Resolution Optical Flow》2022CVPR 光流
weixin_39406427的博客
04-27 1072
Zihua Zheng ； Ni Nie Megvii ；Tencent 这篇文章，作者提出了一个基于PatchMatch方法的高分辨率光流估计方法。 可以以低内存占用率的状态下高精度的获取光流，引入PatchMatch就是为了对抗RAFT的4D All Paired Correlation Volume的高冗余计算。 下图就是4D All Paired Correlation Volume 进一步讲，为了进一步压缩计算量，作者优化了PatchMatch方法中的Propagation...
CVPR 2016 摘要阅读
dlaicxf的博客
10-28 1万+
为了说明看过CVPR2016全部文章的摘要，总结一下，摘要只保留了创新点部分。 ORAL SESSION Image Captioning and Question Answering Monday, June 27th, 9:00AM - 10:05AM. These papers will also be presented at the following poste
视频目标检测FGFA框架：Flow-Guided Feature Aggregation研究
资源摘要信息:"Flow-Guided-Feature-Aggregation研究基于视频的目标检测FGFA框架.zip" 目标检测是计算机视觉领域中一项基础且关键的技术，它涉及从图像中识别出一个或多个目标的位置并分类。目标检测的挑战在于不同...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值