softmax regression 推导

$$P(y^{(i)} = k | x^{(i)} ; \theta) = \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) }$$

似然函数

$$L=\prod_{i=1}^{M}\prod_{k=1}^{K} P(^{y^{(i)} } =k|x^{(i)};\theta )^{1\{y^{ (i)}{=k} \} }$$

对数损失函数为:

$$\begin{align} J(\theta) = - \left[ \sum_{i=1}^{m} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right] \end{align}$$

1{⋅} is the ”‘indicator function,”’ so that 1{a true statement}=1, and 1{a false statement}=0.

现在对对数损失函数求偏导

$$\nabla_{\theta^{(n)}}J(\theta)= \sum_{i=1}^{m} y^{(i)} \frac{\partial P(y^{(i)}=n|x^{i};\theta)}{\partial \theta^{(n)}} + \sum_{k=1,k\ne n}^{K} y^{(i)} \frac{\partial P(y^{(i)}=k|x^{i};\theta)}{\partial \theta^{(n)} }$$
其中,
$$P(y^{(i)}=n|x^{i};\theta)=\log \frac{\exp(\theta^{(n)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}$$

$$P(y^{(i)}=k|x^{i};\theta) = \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})} \\ $$

$$\frac { \partial P(y^{(i)}=n|x^{i};\theta) }{\partial \theta^{(n)} }\\ = \frac{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}{\exp(\theta^{(n)\top} x^{(i)})} *\left( \frac{ \exp(\theta^{(n)\top} x^{(i)})* x^{(i)}}{ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) } -\frac{ \exp(\theta^{(n)\top} x^{(i)}) *\exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \left[ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) \right]^2 }\right)\\ =x^{(i)}-\frac{ \exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\\ =x^{(i)}\left(1-P(y^{(i)}=n|x^{i};\theta)\right)$$

另外一个，

$$\frac{\partial P(y^{(i)}=k|x^{i};\theta) } {\partial \theta^{(n)} }\\ = \frac{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}{\exp(\theta^{(k)\top} x^{(i)})} \left( -\frac{ \exp(\theta^{(k)\top} x^{(i)}) *\exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \left[ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) \right]^2 } \right)\\ =-\frac{ \exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) }\\ =-P(y^{(i)}=n|x^{i};\theta)x^{(i)}$$
$$\begin{align} \nabla_{\theta^{(k)}} J(\theta) = - \sum_{i=1}^{m}{ \left[ x^{(i)} \left( 1\{ y^{(i)} = k\} - P(y^{(i)} = k | x^{(i)}; \theta) \right) \right] } \end{align}$$