two-stream fusion network

Two-stream network

文章：

Convolutional Two-Stream Network Fusion for Video Action Recognition

网络融合：

Conv fusion

$${\mathbf{y}}^{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}=f^{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}\left({\mathbf{x}}^a,{\mathbf{x}}^b\right)$$

$${\mathbf{y}}^{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}={\mathbf{y}}^{\mathrm{c}\mathrm{a}\mathrm{t}}\ast \mathbf{f}+b$$

$$\mathbf{f}\in {\mathbb{R}}^{1\times 1\times 2D\times D}$$

先叠加再卷积，将通道变更为D。

Concatenation fusion

$${\mathbf{y}}^{\mathrm{c}\mathrm{a}\mathrm{t}}=f^{\mathrm{c}\mathrm{a}\mathrm{t}}\left({\mathbf{x}}^a,{\mathbf{x}}^b\right)$$

$$y_{i,j,2d}^{\mathrm{c}\mathrm{a}\mathrm{t}}=x_{i,j,d}^a{y}_{i,j,2d-1}^{\mathrm{c}\mathrm{a}\mathrm{t}}=x_{i,j,d}^b$$

$$\mathbf{y}\in {\mathbb{R}}^{H\times W\times 2D}$$

Sum fusion

$${\mathbf{y}}^{\mathrm{s}\mathrm{u}\mathrm{m}}=f^{\mathrm{s}\mathrm{u}\mathrm{m}}\left({\mathbf{x}}^a,{\mathbf{x}}^b\right)$$

$$y_{i,j,d}^{\mathrm{s}\mathrm{u}\mathrm{m}}=x_{i,j,d}^a+x_{i,j,d}^b$$

$$\mathbf{y}\in {\mathbb{R}}^{H\times W\times D}$$

Max fusion

$${\mathbf{y}}^{\max }=f^{\max }\left({\mathbf{x}}^a,{\mathbf{x}}^b\right)$$

$$y_{i,j,d}^{\max }=\max \left\{x_{i,j,d}^a,x_{i,j,d}^b\right\}$$

Bilinear fusion

$${\mathbf{y}}^{\mathrm{b}\mathrm{i}\mathrm{l}}=f^{\mathrm{b}\mathrm{i}\mathrm{l}}\left({\mathbf{x}}^a,{\mathbf{x}}^b\right)$$

$${\mathbf{y}}^{\mathrm{b}\mathrm{i}\mathrm{l}}=\sum _{i=1}^H\sum _{j=1}^W{\mathbf{x}}_{i,j}^{a\top }{\mathbf{x}}_{i,j}^b$$

框架