flow model

• https://www.youtube.com/watch?v=uXY18nzdSsM
• component by component(auto regresive)的生成模型存在的问题(生成的顺序，生成的速度慢)；variational auto-encoder存在的问题(optimize a maxihood lower bound，是一个近似)；generative adaversarial network(unstable training)
• generator:
  – $G$一个Network，定义了一个probability distribution $p_G$，现在有一个从正态分布中采样的$z$，经过$G$得到$x=G(z)$，$x$满足$p_G(x)$的分布，希望这个分布和$p_{data}(x)$越接近越好， { x 1 , x 2 , ⋯   , x m } ∈ p d a t a ( x ) \{x^1,x^2,\cdots,x^m\}\in p_{data}(x) {x1,x2,⋯,xm}∈pdata​(x)；
  – 如何使得两个分布越相近越好，一般是maximize likelihood $G^{\ast }=argma{x}_G{\sum }_{i=1}^m\log p_G(x^i)$，使得$x^i$被$G$产生的几率越大越好；
  – flow model直接优化上式；
• math background
  – jacobian: $x=f(z),z=\left[\begin{array}{c}{z}_1\\ z_2\end{array}\right],x=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$，jacobian的定义是$J_f=\left[\begin{array}{cc}\partial x_1/\partial z_1& \partial x_1/\partial z_2\\ \partial x_2/\partial z_1& \partial x_2/\partial z_2\end{array}\right]$,此时$z=f^{-1}(x)$,$J_{f^{-1}}=\left[\begin{array}{cc}\partial z_1/\partial x_1& \partial z_1/\partial x_2\\ \partial z_2/\partial x_1& \partial z_2/\partial x_2\end{array}\right]$;两个jacobian $J_f\ast J_{f^{-1}}=I$，两者互逆
  – determinant：行列式det(A),A是一个方阵，$det(A)=1/det(A^{-1})$；行列式可以理解为高维空间中的体积的概念；
  – change of variable: 假设现在有一个正态分布$\pi (z)$，$x=f(z)$，$x$满足分布$p(x)$，$p(x^{\prime })\Delta x=\pi (z^{\prime })\Delta z\to p(x^{\prime })=\pi (z^{\prime })\frac{\Delta z}{\Delta x}\to p(x^{\prime })=\pi (z^{\prime })\left|\frac{dz}{dx}\right|$，接下来扩展到二维，两块的面积相等，$p(x^{\prime })\left|det\left[\begin{array}{cc}\Delta x_{11}& \Delta x_{21}\\ \Delta x_{12}& \Delta x_{22}\end{array}\right]\right|=\pi (z^{\prime })\Delta z_1\Delta z_2$,其中$\Delta x_{12},\Delta x_{22}$分别是$z_2$改变的时候$x_1,x_2$的改变量，$\Delta x_{11},\Delta x_{21}$是当$z_1$改变的时候$x_1,x_2$的改变量；接下来进行整理：$\pi (z^{\prime })=p(x^{\prime })\left|\frac{1}{\Delta z_1\Delta z_2}det\left[\begin{array}{cc}\Delta x_{11}& \Delta x_{21}\\ \Delta x_{12}& \Delta x_{22}\end{array}\right]\right|=p(x^{\prime })\left|det\left[\begin{array}{cc}\Delta x_{11}/\Delta z_1& \Delta x_{21}/\Delta z_1\\ \Delta x_{12}/\Delta z_2& \Delta x_{22}/\Delta z_2\end{array}\right]\right|=p(x^{\prime })\left|det\left[\begin{array}{cc}\partial x_1/\partial z_1& \partial x_2/\partial z_1\\ \partial x_1/\partial z_2& \partial x_2/\partial z_2\end{array}\right]\right|=p(x^{\prime })\left|det\left[\begin{array}{cc}\partial x_1/\partial z_1& \partial x_1/\partial z_2\\ \partial x_2/\partial z_1& \partial x_2/\partial z_2\end{array}\right]\right|=p(x^{\prime })|det(J_f)|$,也可以得到$p(x^{\prime })=\pi (z^{\prime })|det(J_{f^{-1}})|$
  
• flow model：原目标是maximize likelihood $G^{\ast }=argma{x}_G{\sum }_{i=1}^m\log p_G(x^i)$，而$p_G(x^i)=\pi (z^i)|det(J_{G^{-1}})|,z^i=G^{-1}(x^i)$，有$\log p_G(x^i)=\log \pi (G^{-1}(x^i))+\log |det(J_{G^{-1}})|$；需要计算$det(J_G),G^{-1}$，为了保证invertible，输入和输出保持维度相同；
  $p_1(x^i)=\pi (z^i)(|det(J_{G_1^{-1}})|)$
  $p_2(x^i)=\pi (z^i)(|det(J_{G_1^{-1}})|)(|det(J_{G_2^{-1}})|)$
  $\cdots$
  $p_K(x^i)=\pi (z^i)(|det(J_{G_1^{-1}})|)\cdots (|det(J_{G_K^{-1}})|)$
  $\log p_K(x^i)=\log \pi (z^i)+{\sum }_{h=1}^K\log |det(J_{G_K^{-1}})|,z^i=G_1^{-1}(\cdots G_K^{-1}(x^i))$
  可以看到上式只有$G^{-1}$
• coupling layer:nice nvp glow
  前向：
  
  逆向
  接下来计算Jacobian
  
  上面是一层的情况，接下来叠起来：