逻辑回归

理论

逻辑函数：$$g(z)=\frac{1}{1+e^{-z}}\text{\hspace{1em}(1)}$$

逻辑函数可视化：

import matplotlib.pyplot as plt
import numpy as np
import math
e = math.e
x = np.linspace(-10,10,1e6)
y = 1 / (1 + np.exp(-x))

plt.plot(x, y)
plt.show()

假设函数：$$h_{\theta }(x)=g({\theta }^{T}X)=\frac{1}{1+e^{-{\theta }^{T}X}}\text{\hspace{1em}(2)}$$

代价函数：$$J(\theta )=\frac{1}{m}\sum _{i=1}^{m}Cost(h_{\theta }(x^{(i)}),y^{(i)})\text{\hspace{1em}(3)}$$
其中，$$Cost(h_{\theta }(x^{(i)}),y^{(i)})=\{\begin{array}{ll}-log(h_{\theta }(x)),& \text{if y=1}\\ -log(1-h_{\theta }(x)),& \text{if y=0}\end{array}\text{\hspace{1em}(4)}$$

$$J\left(\theta \right)=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\log \left(h_{\theta }\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)]\text{\hspace{1em}(5)}$$

其函数图像为：

从图中可以看出:

• $y=1$，当预测值$h_{\theta }(x)=1$时，代价函数$Cost$的值为0，这是我们想要的（模型预测完全正确时，代价达到最小）。当预测值离1越远，其代价函数越大，这也是我们想要的
• 同理$y=0$，当预测值=0时，代价函数达到最小值；预测值离0越远，其代价函数越大。

代价函数推导过程(采用极大似然估计)：

假设函数$h_{\theta }(x)$表示预测结果为1的概率，则：
$$\begin{gathered} P(y=1|x;\theta )=h_{\theta }(x) \\ P(y=0|x;\theta )=1-h_{\theta }(x)\text{\hspace{1em}(6)} \end{gathered}$$

将公式6合并为一个公式：
$$P(y|x;\theta )=h_{\theta }(x{)}^y\ast (1-h_{\theta }(x){)}^{1-y}\text{\hspace{1em}(7)}$$

取似然函数：
$$L(\theta )=\prod _{i=1}^{m}P(y^{(i)}|x^{(i)};\theta )=\prod _{i=1}^m(h_{\theta }(x^{(i)}){)}^{y^{(i)}}(1-h_{\theta }(x^{(i)}){)}^{1-y^{(i)}}\text{\hspace{1em}(8)}$$

对数似然函数：
$$l_{\theta }=log(L(\theta ))=\sum _{i=1}^m[y^{(i)}\log \left(h_{\theta }\left(x^{(i)}\right)\right)+\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)]\text{\hspace{1em}(9)}$$

最大似然估计是取使得似然函数最大化的$\theta$，令损失函数$J(\theta )=-\frac{1}{m}l(\theta )$，则最大化的$l(\theta )$即为最小化的$J(\theta )$

梯度下降法迭代公式：
$${\theta }_j={\theta }_j-\alpha (\frac{\partial }{\partial {\theta }_j})J(\theta )={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}\text{\hspace{1em}(10)}$$

矩阵形式：
$$\theta =\theta -\alpha \frac{1}{m}{x}^T(g(x\theta )-y)\text{\hspace{1em}(11)}$$

推导如下：

带惩罚项的逻辑回归

$$J\left(\theta \right)=\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log \left(h_{\theta }\left(x^{(i)}\right)\right)-\left(1-y^{(i)}\right)\log \left(1-h_{\theta }\left(x^{(i)}\right)\right)]+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2\text{\hspace{1em}(12)}$$

重复直至收敛：

${\theta }_0:={\theta }_0-a\frac{1}{m}\sum _{i=1}^m((h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)})$

${\theta }_j:={\theta }_j-a[\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{\left(i\right)}+\frac{\lambda }{m}{\theta }_j]$

$j=1,2,...n$

Python实现

import numpy as np

X = np.array([[1, 2], [3, 2], [1, 3], [2, 3], [3, 3], [3, 4], [10, 11], [9, 10], [12, 13], [14, 14], [13, 12]])
y = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0])
n_samples, n_features = X.shape

X = np.concatenate((np.ones(n_samples).reshape((n_samples, 1)), X), axis=1)
y = y.reshape((n_samples, 1))

max_iter = 1e4  # 最大迭代次数
epsilon = 1e-4  # θ迭代前后变化最大误差不能超过epsilon
theta = np.zeros((n_features + 1, 1))  # 初始化theta
alpha = 0.0001

for iter in range(int(max_iter)):
    theta_next = theta - alpha * (X.T) @ (1 / (1 + np.exp(-X@theta)) - y) / n_samples
    print(theta_next)
    if np.abs(theta - theta_next).sum() < epsilon:
        theta = theta_next
        print("merge")
        break
    theta = theta_next
else:
    print("get the max_iter, stop iter.")