UFLDL Softmax Regression 推导

Softmax Regression

Logistic regression通常用来解决二分类问题，解决多分类问题通常就要用到softmax regression。给定一个样例x，假设我们想要计算出x属于k$\in\left\{1,2,3...K\right\}$的概率$P\left\{y=k|x\right\}$可以使用如下的公式：

$$h_\theta(x)= \left[ \begin{matrix} P(y=1|x;\theta) \\ P(y=2|x;\theta) \\ \vdots \\ P(y=K|x;\theta) \\ \end{matrix} \right]=\frac{1}{\sum_{j=1}^{K}exp(\theta^{(j)T}x)} \left[ \begin{matrix} exp(\theta^{(1)T}x) \\ exp(\theta^{(1)T}x) \\ \vdots \\ exp(\theta^{(1)T}x) \\ \end{matrix} \right]$$
其中 $\theta^1,\theta^2...\theta^k\in R^n$是模型的参数，其中 $\frac{1}{\sum_{j=1}^kexp(\theta^{(j)T}x)}$是标准化项，使得所有类的概率相加等于1， $\theta$是一个n*k的系数矩阵，具有如下形式：
$$\left[ \begin{matrix} \theta^{1} & \theta^{2} & \cdots & \theta^{K} \end{matrix} \right]$$

Cost function

在cost function中使用指示函数，1{true} = 1,1{false} = 0,cost function具有如下形式：

$$J(\theta) = -[\sum_{i=1}^{m}\sum_{k=1}^K1\left\{y^i=k\right\}log\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^Kexp(\theta^{(j)T}x^i)}]$$
对每一个 $\theta^l$求导， $l\in\left\{1,2...K\right\}$
$$\frac{\partial J(\theta)}{\partial \theta^l} = -\sum_{i=1}^{m}[\frac{\partial \sum_{k=1}^K1\left\{y^i=k\right\}log\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^Kexp(\theta^{(j)T}x^i)}}{\partial \theta^l}]$$
当l=k时
$$\begin{align*} &\frac{\partial 1\left\{y^i=k\right\}log\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^Kexp(\theta^{(j)T}x^i)}}{\partial \theta^l}\\ &=\left\{y^i=k\right\}\frac{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)}{exp(\theta^{(k)T}x^i)}*\\ &\frac{x^i*exp(\theta^{(k)T}x^i)*{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)}-{x^i*exp(\theta^{(k)T}x^i)*{exp(\theta^{(k)T}x^i)}}}{{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)}^2}\\ &=x^i*(1-\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)})\\ &=x^i*(1-P(y^i=k|x^i;\theta)\\ &=x^i*(1\left\{y^i=k\right\}-P(y^i=k|x^i;\theta)) \end{align*}$$
当 $l\not=k$时
$$\begin{align*} &\frac{\partial 1\left\{y^i=k\right\}log\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^Kexp(\theta^{(j)T}x^i)}}{\partial \theta^l}\\ &=\left\{y^i=k\right\}\frac{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)}{exp(\theta^{(k)T}x^i)}*\frac{0-{x^i*exp(\theta^{(k)T}x^i)*{exp(\theta^{(k)T}x^i)}}}{{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)}^2}\\ &=x^i*(0-\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)})\\ &=x^i*(0-P(y^i=k|x^i;\theta))\\ &=x^i*(1\left\{y^i=k\right\}-P(y^i=k|x^i;\theta)) \end{align*}$$
所以
$$\frac{\partial J(\theta)}{\partial \theta^l} = -\sum_{i=1}^{m}[x^i*(1\left\{y^i=k\right\}-P(y^i=k|x^i;\theta))]$$

Softmax Regression参数特点

Softmax Regression回归所求得参数是一个冗余的参数集，把得到的每一个参数$\theta_l$同时减去$\varphi$，并不影响预测结果，此时$\theta_l$都变成了$\theta_l - \varphi,l\in\left\{1,2,3..K\right\}$，数学推导如下：

$$\begin{align*} P(y^i=k|x^i;\theta)& = \frac{exp((\theta^{(k)}-\varphi)^Tx^i)}{\sum_{j=1}^{K}exp((\theta^{j}-\varphi)x^i)}\\ &=\frac{exp(\theta^{(k)T}x^i)exp(\varphi^Tx^i)}{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)exp(\varphi^Tx^i)}\\ &=\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^{K}exp(\theta^{(j)T}x^i)} \end{align*}$$
为了避免求得的是一个冗余的参数集，可以考虑在cost function中加入一个正则化项，加入正则项后的cost function和导数表达式如下：
$$\begin{align} J(\theta) = -[\sum_{i=1}^{m}\sum_{k=1}^K1\left\{y^i=k\right\}log\frac{exp(\theta^{(k)T}x^i)}{\sum_{j=1}^Kexp(\theta^{(j)T}x^i)}] + \lambda/2\sum_{i=1}^{k}\sum_{j=1}^{m}\theta_{ij}^2 \end{align}$$
$$\begin{align} \frac{\partial J(\theta)}{\partial \theta^l} = -\sum_{i=1}^{m}[x^i*(1\left\{y^i=k\right\}-P(y^i=k|x^i;\theta))] + \lambda\theta_l \end{align}$$
最后exercise练习如下:

  yLabel = full(sparse(y, 1:m, 1));   
  yLabel = yLabel(1:num_classes-1,:);  
  M = exp(theta'*X);
  p = bsxfun(@rdivide,M,sum(M));
  f = - yLabel(:)'*log(p(:));
  g = - X*(yLabel - p)';
  g = g(:,1:num_classes - 1);