逻辑回归

1. 逻辑回归模型

逻辑回归将线性回归的预测值转换为0/1值，通常采用sigmoid函数$g(z)=\frac{1}{1+e^{-z}}$进行转换：
$$h(x)=\frac{1}{1+e^{-({\omega }^{T}x+b)}}$$
即
$$\begin{gathered} \begin{array}{rl}P(y=1|x)& =\frac{1}{1+e^{-({\omega }^{T}x+b)}}\\ & =\frac{e^{{\omega }^{T}x+b}}{1+e^{{\omega }^{T}x+b}}\end{array} \\ P(y=0|x)=\frac{1}{1+e^{{\omega }^{T}x+b}} \end{gathered}$$

2. 损失函数

最大化对数似然函数
$$\begin{array}{rl}L(\omega ,b)& =\sum _{i=1}^m\ln P(y_i|x_i;\omega ,b)\\ & =\sum _{i=1}^m\ln (y_i{P}_1(x_i;\omega ,b)+(1-y_i)P_0(x_i;\omega ,b))\\ & =\sum _{i=1}^m(y_i\ln h(x_i)+(1-y_i)\ln (1-h(x_i)))\end{array}$$
损失函数为
$$E(\omega ,b)=-\sum _{i=1}^m(y_i\ln h(x_i)+(1-y_i)\ln (1-h(x_i)))$$

3. 损失函数求解

3.1 梯度下降法

令$\hat{\omega }=(\omega ;b)$
$$\begin{array}{rl}E(\hat{\omega },b)& =-\sum _{i=1}^m(y_i\ln h(x_i)+(1-y_i)\ln (1-h(x_i)))\\ & =-\sum _{i=1}^m(y_i({\omega }^T{x}_i+b)-\ln (1+e^{{\omega }^T{x}_i+b}))\\ & =-\sum _{i=1}^m(y_i{\hat{\omega }}^T{x}_i-\ln (1+e^{{\hat{\omega }}^T{x}_i}))\\ \frac{\partial E}{\partial \hat{\omega }}& =-\sum _{i=1}^m(y_{i}x-\frac{e^{{\hat{\omega }}^{T}x}}{1+e^{{\hat{\omega }}^{T}x}}x)\\ & =\sum _{i=1}^m(h(x_i)-y_i)x_i\\ & =X^T(h(X)-Y)\end{array}$$
其中$X$为为$m\times n$维的矩阵，$m$代表样本的个数，$n$代表样本的特征数。参数更新公式为：
$$\hat{\omega }=\hat{\omega }-\alpha X^T(h(X)-Y)$$

3.2 牛顿法

4. 为什么sigmoid输出代表概率

参见广义线性模型推导过程

5. 共线性问题