Softmax回归 — Softmax Regression

之前讲过，对于二分类问题我们可以采用逻辑回归。逻辑回归通过$\text{Logit Function}$将$x^i$映射到$(0,1)$的一个区间，可解释为事件$x^i$发生的概率$p(x_i;\theta )=\frac{1}{1+e^{-{\theta }^T{x}_i}}$。而对于多分类问题 y i ∈ { 1 , 2 , … , K } y^i \in \{1, 2, \dots, K\} yi∈{1,2,…,K}，$\text{Softmax Regression}$采用同样的思路，同样是将$x^i$映射到$(0-1)$的区间上，计算$x^i$发生的概率。不过，$p(x_i;\theta )$的定义稍有不同：
$$p(x_i;\theta )=Pr(y^i=k|x^i;\theta )=\frac{e^{{{\theta }^{(k)}}^T{x}^i}}{{\sum }_{j=1}^K{e}^{{{\theta }^{(j)}}^T{x}^i}}$$

其中，分子表示数据$x^i$属于类$k$的概率，分母表示其属于不同类的概率之和，$p(x_i;\theta )\in (0,1)$。为加快程序运行速度，在代码实现中通常会向量化上式：
$$p(x_i;\theta )=\left[\begin{array}{c}Pr(y^i=1|x^i;\theta )\\ Pr(y^i=2|x^i;\theta )\\ ⋮\\ Pr(y^i=K|x^i;\theta )\end{array}\right]=\frac{1}{{\sum }_{j=1}^K{e}^{{{\theta }^{(j)}}^T{x}^i}}\left[\begin{array}{c}{e}^{{{\theta }^{(1)}}^T{x}^i}\\ e^{{{\theta }^{(2)}}^T{x}^i}\\ ⋮\\ e^{{{\theta }^{(K)}}^T{x}^i}\end{array}\right]$$

$\text{Softmax Regression}$最大似然函数为：
L ( θ ) = ∏ i = 1 m p ( x i ; θ ) 1 { y i = k } L(\theta) = \prod\limits_{i=1}^m p(x_i; \theta)^{1\{y^i=k\}} L(θ)=i=1∏m​p(xi​;θ)1{yi=k}

其中， 1 { ⋅ } 1\{ \cdot \} 1{⋅}是指示函数， 1 { T r u e } = 1 ; 1 { F a l s e } = 0 1\{ True \} = 1; 1\{ False\} = 0 1{True}=1;1{False}=0。对数似然函数为：
l ( θ ) = log ⁡ L ( θ ) = ∑ i = 1 m 1 { y i = k } log ⁡ e θ ( k ) T x i ∑ j = 1 K e θ ( j ) T x i l(\theta) = \log L(\theta) = \sum\limits_{i=1}^m 1\{ y^i=k \} \log \frac {e^{{\theta^{(k)}}^T x^i}}{\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}} l(θ)=logL(θ)=i=1∑m​1{yi=k}log∑j=1K​eθ(j)Txieθ(k)Txi​

成本函数$J(\theta )$定义为：
J ( θ ) = − l ( θ ) = − ∑ i = 1 m 1 { y i = k } log ⁡ e θ ( k ) T x i ∑ j = 1 K e θ ( j ) T x i J(\theta) = - l(\theta) = - \sum\limits_{i=1}^m 1\{ y^i=k \} \log \frac {e^{{\theta^{(k)}}^T x^i}}{\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}} J(θ)=−l(θ)=−i=1∑m​1{yi=k}log∑j=1K​eθ(j)Txieθ(k)Txi​

$J(\theta )$对${\theta }^{(k)}$求偏导：
∂ J ( θ ) ∂ θ ( k ) = ∂ ∂ θ ( k ) [ − ∑ i = 1 m 1 { y i = k } log ⁡ e θ ( k ) T x i ∑ j = 1 K e θ ( j ) T x i ] = ∂ ∂ θ ( k ) [ − ∑ i = 1 m 1 { y i = k } ( log ⁡ e θ ( k ) T x i − log ⁡ ∑ j = 1 K e θ ( j ) T x i ) ] = − ∑ i = 1 m ( 1 { y i = k } x i − e θ ( j ) T x i ∑ j = 1 K e θ ( j ) T x i x i ) = − ∑ i = 1 m x i ( 1 { y i = k } − e θ ( j ) T x i ∑ j = 1 K e θ ( j ) T x i ) = − ∑ i = 1 m x i ( 1 { y i = k } − P r ( y i = k ∣ x i ; θ ) ) \begin{aligned} & \frac {\partial J(\theta)}{\partial \theta^{(k)}} = \frac {\partial}{\partial \theta^{(k)}} [- \sum\limits_{i=1}^m 1\{ y^i=k \} \log \frac {e^{{\theta^{(k)}}^T x^i}}{\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}}] \\ & = \frac {\partial}{\partial \theta^{(k)}} [- \sum\limits_{i=1}^m 1\{ y^i=k \} (\log e^{{\theta^{(k)}}^T x^i} - \log {\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}})] \\ & = - \sum\limits_{i=1}^m (1\{ y^i=k \} x^i - \frac {e^{{\theta^{(j)}}^T x^i}}{\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}} x^i) \\ & = - \sum\limits_{i=1}^m x^i(1\{ y^i=k \} - \frac {e^{{\theta^{(j)}}^T x^i}}{\sum\nolimits_{j=1}^K e^{{\theta^{(j)}}^T x^i}}) \\ & = - \sum\limits_{i=1}^m x^i(1\{ y^i=k \} - Pr(y^i = k \vert x^i; \theta)) \end{aligned} ​∂θ(k)∂J(θ)​=∂θ(k)∂​[−i=1∑m​1{yi=k}log∑j=1K​eθ(j)Txieθ(k)Txi​]=∂θ(k)∂​[−i=1∑m​1{yi=k}(logeθ(k)Txi−log∑j=1K​eθ(j)Txi)]=−i=1∑m​(1{yi=k}xi−∑j=1K​eθ(j)Txieθ(j)Txi​xi)=−i=1∑m​xi(1{yi=k}−∑j=1K​eθ(j)Txieθ(j)Txi​)=−i=1∑m​xi(1{yi=k}−Pr(yi=k∣xi;θ))​

最后，通过梯度下降求$J(\theta )$最小化时${\theta }^{(k)}$的近似解：
$${\theta }^{(k)}:={\theta }^{(k)}-\alpha \frac{\partial J(\theta )}{\partial {\theta }^{(k)}}$$

将上式写成向量形式，即：$\theta =\theta -\alpha ▽J(\theta )$。