softmax——笔记

最新推荐文章于 2024-09-28 20:33:52 发布

last_summer_x

最新推荐文章于 2024-09-28 20:33:52 发布

阅读量70

点赞数

分类专栏：笔记文章标签： softmax 机器学习

本文链接：https://blog.csdn.net/last_summer_x/article/details/103103820

版权

笔记专栏收录该内容

3 篇文章 0 订阅

订阅专栏

softmax

用多项式分布建模
参数估计

用多项式分布建模

softmax假设目标变量服从多项式分布
$\begin{aligned} P(y;\eta)&=\prod_{i=1}^{k}\phi_i^{1\{ y=i \}}\\ &=\left( \prod_{i=1}^{k-1}\phi_i^{y=i} \right)\phi_k^{1-\sum_{i=1}^{k-1}{1\{ y=i \}}}\\ &=\left( \prod_{i=1}^{k-1}{(\frac{\phi_i}{\phi_k})}^{1\{ y=i \}} \right)\phi_k\\ &=\exp{\left( \sum_{i=1}^{k-1}\log(\frac{\phi_i}{\phi_k})T(y)_i+\log{\phi_k} \right)} \end{aligned}$
其中
$\begin{aligned} \eta&=\left[ \log{\frac{\phi_1}{\phi_k}},\dots,\frac{\phi_{k-1}}{\phi_k} \right]^T\\ T(1)&=\left[1,0,0,\dots,0 \right]^T\\ T(2)&=\left[0,1,0,\dots,0 \right]^T\\ &\cdots\\ T(k-1)&=\left[0,0,0,\dots,1 \right]^T\\ T(k)&=\left[0,0,0,\dots,0 \right]^T \end{aligned}$
为了方便令
$\eta_i=\log{\frac{\phi_i}{\phi_k}},i=1,\dots,n$
其中 $\eta_k=\log{\frac{\phi_l}{\phi_k}}=0$ ，与逻辑回归的假设一样 $\eta_i=$ ，那么
$\begin{aligned} e^{\eta_i}&=\frac{\phi_i}{\phi_k}\\ \phi_ke^{\eta_i}&=\phi_i\\ \phi_k\sum_{i=1}^{k}{e^{\eta_i}}&=\sum_{i=1}^{k}\phi_i=1\\ \phi_k&=\frac{1}{\sum_{i=1}^{k}{e^{\eta_i}}}\\ \Rightarrow \phi_i&=\phi_ke^{\eta_i}=\frac{e^{\eta_i}}{\sum_{i=1}^{k}{e^{\eta_i}}} \end{aligned}$
和逻辑回归一样，softmax同样假设 $\eta_i=\theta_i^Tx$ ，故
$\begin{aligned} &P(y=i|x;\theta)=\phi_i=\frac{e^{\theta_i^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}}\\ \phi&=\left[ \frac{e^{\theta_1^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}},\dots,\frac{e^{\theta_{k-1}^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}},\frac{e^{\theta_{k}^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}} \right]^T\\ &=\left[ \frac{e^{\theta_1^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}},\dots,\frac{e^{\theta_{k-1}^Tx}}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}},\frac{1}{\sum_{j=1}^{k}{e^{\theta_j^Tx}}} \right]^T \end{aligned}$
则
$y=\argmax_i{P(y=i|x;\theta)}=\argmax_i{\phi_i}$

参数估计

使用对数似然对参数进行估计
$\begin{aligned} \ell\theta)&=\sum_{i=1}^{n}{\log{P(y^{(i)}|x^{(i)};\theta)}}\\ &=\sum_{i=1}^{n}{\log{\prod_{j=1}^{k}{\phi_k^{1\{ y=j \}}}}}\\ &=\sum_{i=1}^{n}{\left( \log\phi \right)^TT(y^{(i)})} \end{aligned}$
对 $\ell\theta)$ 求偏导，对于 $\neq y,i=1,\dots,k-1$
$\frac{\partial \ell \left( \theta \right)}{\partial \theta _i}=\frac{\partial \ell \left( \theta \right)}{\partial \phi _i}\frac{\partial \phi _i}{\partial e^{\theta _{i}^{T}x}}\frac{\partial e^{\theta _{i}^{T}x}}{\partial \theta _i}$
其中 $\phi_i =e^{\theta _{i}^{T}x}/\sum_{j=1}^k{e^{\theta _{j}^{T}x}}$ ，由于 $T(y)_i=0$ ，故 $\ell(\theta)=\log{\phi_i}$ 。当 $\neq y$ 时
$\begin{aligned} \frac{\partial \ell \left( \theta \right)}{\partial \theta _i}&=\frac{1}{\phi _i}\frac{0-e^{\theta _{y}^{T}x}}{\left( \sum_{j=1}^k{e^{\theta _{j}^{T}x}} \right) ^2}e^{\theta _{i}^{T}x}x\\ &=\frac{1}{\phi _i}\phi _i\phi _yx\\ &=\phi _yx \end{aligned}$
当 $i = y$ 时
$\begin{aligned} \frac{\partial \ell \left( \theta \right)}{\partial \theta _i}&=\frac{\partial \ell \left( \theta \right)}{\partial \theta _y}=\frac{1}{\phi _y}\frac{\left( \sum_{j=1}^k{e^{\theta _{j}^{T}x}} \right) -e^{\theta _{y}^{T}x}}{\left( \sum_{j=1}^k{e^{\theta _{j}^{T}x}} \right) ^2}e^{\theta _{y}^{T}x}x\\ &=\frac{1}{\phi _y}\phi _y\left( 1-\phi _y \right) x\\ &=\left( 1-\phi _y \right) x\\ \end{aligned}$
采用随机梯度下降，即每次只使用一个样本 $\left( x^{\left( i \right)},y^{\left( i \right)} \right)$ 计算偏导，对于 $i=1,\cdots,k-1$ ， $\theta$ 的更新策略为
$\theta _i:=\left\{ \begin{array}{c} \theta _i-\alpha \phi _{y^{(i)}}x^{(i)},i\ne y^{(i)}\\ \theta _i-\alpha \left( 1-\phi _{y^{(i)}} \right) x^{(i)},i=y^{(i)}\\ \end{array} \right.$
而 $\theta_k=\vec{0}$