逻辑回归和Softmax回归

1 逻辑回归 (Logistic Regression)

虽然有"回归"两个字, 但实际上是一个二分类模型 (Bernoulli).

1.1 逻辑回归模型

模型假设
$$\begin{array}{rl}\hat{y}=& P_{\theta }(Y=1\mid x)=\frac{1}{1+e^{-{\theta }^{T}x}}=g({\theta }^{T}x)\\ & P_{\theta }(Y=0\mid x)=\frac{e^{-{\theta }^{T}x}}{1+e^{-{\theta }^{T}x}}=1-g({\theta }^{T}x)\end{array}$$

其中，
$$\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ⋮\\ {\theta }_n\end{array}\right],x=\left[\begin{array}{c}1\\ x^{(1)}\\ ⋮\\ x^{(n)}\end{array}\right],g(z)=\frac{1}{1+e^z}.$$

不难看出
$$\log \frac{P_{\theta }(Y=1\mid x)}{P_{\theta }(Y=0\mid x)}={\theta }^{T}x$$

所以, “逻辑回归"又称为"对数几率回归”.

# 定义sigmoid函数
def sigmoid(z):
  return 1/(1+np.exp(-z))

决策函数
$$prediction=\{\begin{array}{ll}1,& P_{\theta }(Y=1\mid x)\ge 0.5\\ 0,& P_{\theta }(Y=1\mid x)<0.5\end{array}=\{\begin{array}{ll}1,& {\theta }^{T}x\ge 0\\ 0,& {\theta }^{T}x<0\end{array}$$

1.2 训练数据集

$$X=\left[\begin{array}{cccc}1& x_1^{(1)}& \cdots & x_1^{(n)}\\ ⋮& ⋮& & ⋮\\ 1& x_m^{(1)}& \cdots & x_m^{(n)}\end{array}\right]=\left[\begin{array}{c}{x}_1^T\\ ⋮\\ x_m^T\end{array}\right],y=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_m\end{array}\right]$$

训练目标：
$$\hat{y}=g(X\theta )\approx y$$

1.3 对数似然函数

在独立同分布假设下，
$$\begin{array}{rl}l(\theta )& =\log P(Y_1=y_1,\cdots ,Y_m=y_m\mid x_1,\cdots ,x_m;\theta )\\ & =\log \prod _{i=1}^m{P}_{\theta }(Y_i=y_i\mid x_i)=\sum _{i=1}^m\log P_{\theta }(Y_i=y_i\mid x_i)\\ & =\sum _{i=1}^m[y_i\log P_{\theta }(Y_i=1\mid x_i)+(1-y_i)\log P_{\theta }(Y_i=0\mid x_i)]\\ & =\sum _{i=1}^m[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)]\end{array}$$

矩阵形式:
$$l(\theta )=y^T\log \hat{y}+(1-y^T)\log (1-\hat{y})$$

1.4 代价函数

损失函数：
L ( y ^ , y ) = { − log ⁡ P ( Y = 1 ∣ x ) , y = 1 − log ⁡ P ( Y = 0 ∣ x ) , y = 0 = − { y log ⁡ y ^ + ( 1 − y ) log ⁡ ( 1 − y ^ ) } \begin{aligned} L(\hat{y},y) &= \begin{cases} -\log P(Y=1\mid x), & y=1 \\ -\log P(Y=0\mid x), & y=0 \end{cases} \\\\ &= -\{y\log{\hat{y}} + (1-y)\log(1-\hat{y})\} \end{aligned} L(y^​,y)​={−logP(Y=1∣x),−logP(Y=0∣x),​y=1y=0​=−{ylogy^​+(1−y)log(1−y^​)}​
代价函数：
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)]+\boxed{\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2}$$

其中方框内为正则化项，上式可以理解为 $J(\theta ):=-\frac{1}{m}l(\theta )+reg(\theta )$.

矩阵形式：
$$J(\theta )=-\frac{1}{m}[y^T\log \hat{y}+(1-y^T)\log (1-\hat{y})]+\boxed{\frac{\lambda }{2m}(\mathring{E}\theta {)}^T(\mathring{E}\theta )}$$

# 计算代价函数
def computeCost(theta, X, y, penalty):=
  first = -1/m * y.T @ np.log(sigmoid(X@theta))
  second = -1/m * (1-y.T)np.log(1-sogmoid(X@theta))
  reg = penalty/(2*m) * thata[1:].T @ theta[1:]
  return first + second + reg

1.5 代价函数的梯度

注意到对 Logistic 函数 $g(z)$ 有
$$g^{\prime }(z)=g(z)(1-g(z))$$
不难得到
$$\begin{array}{rl}\frac{\partial }{\partial {\theta }_0}J(\theta )& =\frac{1}{m}\sum _{i=1}^m({\hat{y}}_i-y_i)x_i^{(0)}\\ \frac{\partial }{\partial {\theta }_j}J(\theta )& =\frac{1}{m}\sum _{i=1}^m({\hat{y}}_i-y_i)x_i^{(j)}+\boxed{\frac{\lambda }{m}\sum _{j=1}^n{\theta }_j},j=1,\cdots ,n\end{array}$$

矩阵形式:
$$\frac{\partial }{\partial \theta }J(\theta )=\frac{1}{m}{X}^T(\hat{y}-y)+\boxed{\frac{\lambda }{m}\mathring{E}\theta }$$

# 计算梯度
def gradient(theta, X, y, penalty):
  first = 1/m * X.T @ (sigmoid(X@theta)-y)
  reg = penalty/m * theta[1:]
  return first + reg

1.6 目标函数

$${\theta }^{\ast }={\mathrm{argmin}}_{\theta }J(\theta )$$

算法 批量梯度下降 (Batch Gradient Descent)

R e p e a t   u n t i l   c o n v e r g e n c e { θ : = θ − α ∂ ∂ θ J ( θ ) } \begin{aligned} &Repeat\ until\ convergence\{\\ &\qquad \theta := \theta - \alpha\frac{\partial}{\partial\theta}J(\theta)\\ &\} \end{aligned} ​Repeat until convergence{θ:=θ−α∂θ∂​J(θ)}​

其中, $\alpha$是学习速率.

# 批量梯度下降
def gradientDescent(theta, X, y, l_rate, penalty, iter):
  cost = np.zeros(n+1)
  cost[0] = computeCost(theta, X, y)
  for k in range(iter):
    theta = theta - l_rate * gradient(theta, X, y, penalty)
    cost[k] = computeCost(theta, X, y)
  return theta, cost

 

2 Softmax Regression

是 Logistic Regression 的推广，用于多分类 (Multinoulli)。

2.1 Softmax Regression 模型

模型假设
$$\hat{y}\equiv \left[\begin{array}{c}{P}_{\theta }(Y=1\mid \mathbf{x})\\ ⋮\\ P_{\theta }(Y=K\mid \mathbf{x})\end{array}\right]=\left[\begin{array}{c}softmax(W^T\mathbf{x}+{\mathbf{b}}^T{)}_1\\ ⋮\\ softmax(W^T\mathbf{x}+{\mathbf{b}}^T{)}_K\end{array}\right]$$
其中，
$$\begin{array}{rl}{W}^T=\left[\begin{array}{c}{w}^{(1)T}\\ ⋮\\ w^{(K)T}\end{array}\right]& =\left[\begin{array}{ccc}{w}_1^{(1)}& \cdots & w_n^{(1)}\\ ⋮& \ddots & ⋮\\ w_1^{(K)}& \cdots & w_n^{(K)}\end{array}\right],\mathbf{x}=\left[\begin{array}{c}{x}^{(1)}\\ ⋮\\ x^{(n)}\end{array}\right],{\mathbf{b}}^T=\left[\begin{array}{c}{b}^{(1)}\\ ⋮\\ b^{(K)}\end{array}\right],\\ & \\ & softmax(\mathbf{z}{)}_k=\frac{e^{z^{(k)}}}{{\sum }_j{e}^{z^{(j)}}},k=1,\cdots ,K\end{array}$$
决策函数
$$prediction=\arg \underset{k}{\max }{\hat{y}}^{(k)}$$

2.2 训练数据集

$$X=\left[\begin{array}{c}{\mathbf{x}}_1^T\\ ⋮\\ {\mathbf{x}}_m^T\end{array}\right]=\left[\begin{array}{ccc}{x}_1^{(1)}& \cdots & x_1^{(n)}\\ ⋮& & ⋮\\ x_m^{(1)}& \cdots & x_m^{(n)}\end{array}\right],Y=\left[\begin{array}{c}{\mathbf{y}}_1^T\\ ⋮\\ {\mathbf{y}}_m^T\end{array}\right]=\left[\begin{array}{ccc}{y}_1^{(1)}& \cdots & y_1^{(K)}\\ ⋮& & ⋮\\ y_m^{(1)}& \cdots & y_m^{(K)}\end{array}\right]$$

记
$$Z=XW+b$$
训练目标：
$$\hat{Y}=g(Z)\approx Y$$

2.3 对数似然函数

在独立同分布假设下，
$$\begin{array}{rl}l(\Theta )& =\log P(Y=\mathbf{y}\mid \mathbf{x},\Theta )\\ & =\log \prod _{i=1}^m{P}_{\Theta }(Y_i={\mathbf{y}}_i\mid {\mathbf{x}}_i)=\sum _{i=1}^m\log P_{\Theta }(Y_i={\mathbf{y}}_i\mid {\mathbf{x}}_i)\\ & =\sum _{i=1}^m\sum _{k=1}^K{y}_i^{(k)}\log {\hat{y}}_i^{(k)}={\mathbf{y}}^T\log \hat{\mathbf{y}}\end{array}$$

2.4 代价函数

每个样本的损失函数：
$$\begin{array}{rl}L(\hat{\mathbf{y}},\mathbf{y})& =-\log P(Y=\mathbf{y}\mid \mathbf{x})=-{\mathbf{y}}^T\log \hat{\mathbf{y}}\end{array}$$
训练集上的代价函数：
$$\begin{array}{rl}J(\Theta )& =\frac{1}{m}\sum _{i=1}^{m}L({\hat{\mathbf{y}}}_i,{\mathbf{y}}_i)+\frac{\lambda }{2m}||W|{|}^2\\ & =-\frac{1}{m}\sum _{i=1}^m{\mathbf{y}}_i^T\log {\hat{\mathbf{y}}}_i+\frac{\lambda }{2m}||W|{|}^2\\ & =-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K{y}_i^{(k)}\log {\hat{y}}_i^{(k)}+\frac{\lambda }{2m}\sum _{j=1}^n\sum _{k=1}^K(w_j^{(k)}{)}^2\end{array}$$
矩阵形式：
$$J(\Theta )=-\frac{1}{m}{1}_{1\times m}(Y\otimes \log \hat{Y})1_{K\times 1}+\frac{\lambda }{2m}||W|{|}^2$$

2.5 代价函数的梯度

Step1
$$\begin{array}{rl}\frac{\partial J(\Theta )}{\partial {\hat{y}}_i^{(j)}}& =-\frac{1}{m}\frac{y_i^{(j)}}{{\hat{y}}_i^{(j)}},i=1,\cdots ,m,j=1,\cdots ,K\end{array}$$

且由 softmax 函数性质知：
$$\frac{\partial {\hat{y}}_i^{(j)}}{\partial z_i^{(k)}(L)}=\{\begin{array}{ll}{\hat{y}}_i^{(j)}(1-{\hat{y}}_i^{(j)}),& j=k\\ -{\hat{y}}_i^{(j)}{\hat{y}}_i^{(k)},& j\ne k\end{array}$$

所以：
$$\begin{array}{rl}\frac{\partial }{\partial z_i^{(k)}(L)}J(\Theta )& =\sum _{j=1}^K\frac{\partial J(\Theta )}{\partial {\hat{y}}_i^{(j)}}\frac{\partial {\hat{y}}_i^{(j)}}{z_i^{(k)}(L)}\\ & =-\frac{1}{m}{y}_i^{(k)}(1-{\hat{y}}_i^{(k)})+\frac{1}{m}\sum _{j\ne k}{y}_i^{(j)}{\hat{y}}_i^{(k)}\\ & =\frac{1}{m}\left({\hat{y}}_i^{(k)}-y_i^{(k)}\right),i=1,\cdots ,m,k=1,\cdots ,K\end{array}$$

矩阵形式：
$$✓\frac{\partial J(\Theta )}{\partial Z(L)}=\frac{1}{m}(\hat{Y}-Y)$$

Step2

不难得到
$$\begin{array}{rl}\frac{\partial }{\partial w_j^{(k)}}J(\Theta )& =\sum _{i=1}^m\frac{\partial J(\Theta )}{\partial z_i^{(k)}}\frac{\partial z_i^{(k)}}{\partial w_j^{(k)}}+\frac{\lambda }{m}{w}_j^{(k)}\\ & =\frac{1}{m}\sum _{i=1}^m(y_i^{(k)}-{\hat{y}}_i^{(k)})x_i^{(j)}+\frac{\lambda }{m}{w}_j^{(k)}\\ & \\ \frac{\partial }{\partial b^{(k)}}J(\Theta )& =\sum _{i=1}^m\frac{\partial J(\Theta )}{\partial z_i^{(k)}}\frac{\partial z_i^{(k)}}{\partial b^{(k)}}=\frac{1}{m}\sum _{i=1}^m(y_i^{(k)}-{\hat{y}}_i^{(k)})\end{array}$$

矩阵形式:
$$\begin{gathered} \frac{\partial }{\partial W}J(\Theta )=\frac{1}{m}{X}^T(Y-\hat{Y})+\frac{\lambda }{m}W \\ \frac{\partial }{\partial b}J(\Theta )=\frac{1}{m}(Y-\hat{Y}) \end{gathered}$$

2.6 目标函数

$${\Theta }^{\ast }={\mathrm{argmin}}_{\Theta }J(\Theta )$$

算法 批量梯度下降 (Batch Gradient Descent)

R e p e a t   u n t i l   c o n v e r g e n c e { W : = W − α ∂ ∂ W J ( Θ ) b : = b − α ∂ ∂ b J ( Θ ) } \begin{aligned} &Repeat\ until\ convergence\{\\ &\qquad W := W - \alpha\frac{\partial}{\partial W}J(\Theta)\\ &\qquad b := b - \alpha\frac{\partial}{\partial b}J(\Theta)\\ &\} \end{aligned} ​Repeat until convergence{W:=W−α∂W∂​J(Θ)b:=b−α∂b∂​J(Θ)}​

其中, $\alpha$是学习速率.