Gumbel 重参数化相关性质证明

Gumbel 的采样过程：

z = a r g m a x i { g i + l o g ( π i ) } , g i = − l o g ( − l o g ( u i ) ) , u i ∼ U ( 0 , 1 ) z=argmax_i \{g_i + log(\pi_i)\}, g_i = -log(-log(u_i)),u_i\sim U(0, 1) z=argmaxi​{gi​+log(πi​)},gi​=−log(−log(ui​)),ui​∼U(0,1)

采样得到的随机变量满足一下分布：

$$g_i\sim Gumble(0,1)(1)$$

$$h_i=g_i+log({\pi }_i)\sim Gumble(log({\pi }_i),1)(2)$$

证明过程：

$$P(u)=P(U\le u)=u,u\in (0,1)$$

$$G=-log(-log(U)),u\in (0,1)$$

$$P(g)=P(G\le g)=P(-log(-log(U))\le g)$$

$$=P(U\le exp(-exp(-g)))$$

$$=exp(-exp(-g))$$

$$P(g)=exp(-exp(-g))$$

$$g_i\sim Gumble(0,1)$$

$$h_i=g_i+log({\pi }_i)\sim Gumble(log({\pi }_i),1)$$

$$P(Z=z)={\pi }_i(3)$$

证明过程：

$$P(Z=z|U_z=u_z)=\prod _{i\ne z}P(H_i<g_z+log({\pi }_z))$$

$$=\prod _{i\ne z}P(G_i+log({\pi }_i)<g_z+log({\pi }_z))$$

$$=\prod _{i\ne z}P(U_i<u_z^{p_i/p_z})$$

$$=\prod _{i\ne z}{u}_z^{p_i/p_z}=u_z^{1/p_z-1}$$

$$P(Z=z)={\int }_0^{1}P(Z=z|U_z=u_z)P(U_z=u_z)d{u}_z$$

$$={\int }_0^1{u}_z^{1/p_z-1}\ast 1\ast d{u}_z$$

$$=\frac{1}{1/p_z}{u}_z^{1/p_z}{|}_0^1$$

$$=p_z$$

$Z_i-Z_j\sim Logistc(log({\pi }_i)-log({\pi }_j),1)(4)$

证明过程：

用$X,Y$分别代表$Z_i,Z_j$.

$$F_x(x)=exp(-exp(-(x-log({\pi }_x))))$$

$$f_X(x)=exp(-(x-log({\pi }_x))-exp(-(x-log({\pi }_x))))$$

$$F_y(y)=exp(-exp(-(y-log({\pi }_y))))$$

$$f_Y(y)=exp(-(y-log({\pi }_y))-exp(-(x-log({\pi }_y))))$$

$$Z=-Y$$

$$F(z)=P(Z<z)=P(-Y<z)=P(Y>-z)=1-P(Y<-z)=1-exp(-exp(-(-z-log({\pi }_y))))$$

$$=1-exp(-exp(z+log({\pi }_y)))$$

简化表达式，令 $a=log({\pi }_x),b=log({\pi }_y)$，则：

$$f_z(z)=F^{\prime }(z)=-exp(-exp(z+b))(-exp(z+b))=exp(z+b-exp(z+b))$$

$$Q=X-Y=X+Z$$

$$f_q(q)={\int }_{-\infty }^{+\infty }f(x,q-x)dx$$

$$={\int }_{-\infty }^{+\infty }{f}_x(x)f_z(q-x)dx$$

$$={\int }_{-\infty }^{+\infty }exp(-(x-a)-exp(-(x-a)))exp(q-x+b-exp(q-x+b))dx$$

令 $x=-log(-log(u))+a，u\in (0,1)$, 则

$$dx=-\frac{1}{-logu}\frac{-1}{u}du=-\frac{1}{ulogu}du$$

继续推导：

$$f_q(q)={\int }_{-\infty }^{+\infty }exp(log(-logu)-exp(log(-logu)))exp(log(-logu)-a+q+b-exp(log(-logu)-a+q+b))dx$$

$$={\int }_0^1(-logu)(u)(-logu)exp(-a+q+b)exp[logu\ast exp(-a+q+b)]\frac{-1}{ulogu}du$$

令 $c=exp(q-(a-b))$，则

$$f_q(q)={\int }_0^1(-logu)(u)(-logu)(c)(u^c)\frac{-1}{ulogu}du$$

$$=-c{\int }_0^1{u}^{c}logudu$$

$$=-c{u}^{c+1}(\frac{logu}{c+1}-\frac{1}{(c+1{)}^2}){|}_0^1$$

$$=-c[-\frac{1}{(1+c{)}^2}-0]$$

$$=\frac{c}{(1+c{)}^2}$$

$$=\frac{exp(q-(a-b))}{(1+exp(q-(a-b){)}^2}$$

$$=\frac{exp(-(q-(a-b)))}{(1+exp(-(q-(a-b)){)}^2}$$

$$Q\sim Logstic(a-b,1)$$