Logistic回归模型推导及手工编程

一、什么是Logistic回归

 Logistic回归不要看它带了“回归”二字，其实它是一个分类模型，一般用于解决二分类问题，比如用于判断是否患癌等等。

二、Logistic回归模型

Sigmoid函数： 也叫Logistic函数，是一种在生物学上常见的s型曲线，$\sigma =\frac{1}{1+e^{-z}}$,图像如下所示：

 通常把$\sigma (z)>0.5$的部分分类至1标签，$\sigma (z)<0.5$则分类至0标签，即：

$$P(Y=1|x)=\sigma ({\omega }^{T}x)=\frac{1}{1+e^{-{\omega }^{T}x}}=\frac{e^{{\omega }^{T}x}}{1+e^{{\omega }^{T}x}}=\pi (x^{(i)})$$
$$P(Y=0|x)=1-P(Y=1|x)=\frac{1}{1+e^{{\omega }^{T}x}}=1-\pi (x^{(i)})$$

假设训练集$T=(x^{(1)},y^{(1)}),\cdot \cdot \cdot ,(x^{(m)},y^{(m)})$， y ( i ) ∈ { 0 , 1 } y_{(i)}\in \{0,1\} y(i)​∈{0,1}
$$x^{(i)}={\left[\begin{array}{ccc}{x}_1^{(i)}& \cdot \cdot \cdot & x_n^{(i)}\end{array}\right]}^T$$

模型： 求$\underset{\omega }{\max }L(\omega )={\prod }_{i=1}^m\pi (x^{(i)}{)}^{y^{(i)}}(1-\pi (x^{(i)}){)}^{1-y^{(i)}}$

推导： $$\begin{array}{rl}\underset{\omega }{\min }L(\omega )& =-ln\prod _{i=1}^m\pi (x^{(i)}{)}^{y^{(i)}}(1-\pi (x^{(i)}){)}^{1-y^{(i)}}\\ & =-\sum _{i=1}^m[y^{(i)}ln(\pi (x^{(i)}))+(1-y^{(i)})\cdot ln(1+e^{{\omega }^T{x}^{(i)}})]\\ & =-\sum _{i=1}^m[y^{(i)}\cdot {\omega }^T{x}^{(i)}-ln(1+e^{{\omega }^T{x}^{(i)}})]\end{array}$$
接着采用梯度下降法求解：
$$\begin{array}{rl}\frac{\partial L(\omega )}{\partial \omega }& =-\sum _{i=1}^m[y^{(i)}\cdot x^{(i)}-\frac{1}{1+e^{{\omega }^{T}x}}\cdot e^{{\omega }^T{x}^{(i)}}\cdot x^{(i)}]\\ & =-\sum _{i=1}^m[y^{(i)}-\frac{e^{{\omega }^{T}x}}{1+e^{{\omega }^{T}x}}]x^{(i)}\end{array}$$
算法步骤：
Step1: 初始化$k,\epsilon ,\alpha ,MaxN,{\omega }_k$
Step2: 任选样本$(x^{(i)},y^{(i)})$，计算$d_k=-[y^{(i)}-\frac{e^{{\omega }^{T}x}}{1+e^{{\omega }^{T}x}}]x^{(i)}$
Step3: ${\omega }_{k+1}:={\omega }_k-\alpha d_k$
Step4: $k=k+1$,若$||d_k||<\epsilon$或$k>MaxN$,输出$\omega$，否则转回Step2

三、正则化

  为了防止过拟合，添加正则项是一个不错的选择。模型即变为
$$L(\omega )=-\sum _{i=1}^m[y^{(i)}\cdot {\omega }^T{x}^{(i)}-ln(1+e^{{\omega }^T{x}^{(i)}})]+\lambda ||\omega |{|}^2$$
求导过程只需要在求出$$\frac{\partial ||\omega |{|}^2}{\partial \omega }=2\omega$$
最后求导结果为
$$\begin{array}{r}\frac{\partial L(\omega )}{\partial \omega }=-\sum _{i=1}^m[y^{(i)}-\frac{e^{{\omega }^{T}x}}{1+e^{{\omega }^{T}x}}]x^{(i)}+2\omega \end{array}$$

四、多分类问题

 由于Logistic模型用于解决二分类问题，因此在手工编程中若遇到多分类问题时，可以通过多次进行二分类来解决。下面代码解决的便是一个三分类问题。

import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import  MinMaxScaler
from sklearn.model_selection import train_test_split

def load():  # 数据加载,处理
    iris = load_iris()
    scaler = MinMaxScaler()
    x = scaler.fit_transform(iris.data[:,:])
    ones = np.ones(x.shape[0])
    X = np.insert(x, 0, values=ones, axis=1)
    y = iris.target
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
    df_train = pd.concat([pd.DataFrame(X_train),pd.DataFrame(y_train,columns=['Species'])],axis = 1)
    df_test = pd.concat([pd.DataFrame(X_test), pd.DataFrame(y_test,columns=['Species'])], axis=1)
    m,n = X.shape #150,5
    cate = set(y)
    return df_train,df_test,n,cate

def Weight():  # 创造w
    _,_,n,cate = load()
    w = np.ones((n, 1))
    return w

def data_spilt1(df_train):
    X = np.array(df_train.iloc[:,:5])
    Y = np.array(df_train.iloc[:,-1])
    Y[Y == 2] = 1
    return X,Y

def data_spilt2(df_train):
    df_train.drop(df_train.index[(df_train['Species'] == 0)], inplace=True)
    X = np.array(df_train.iloc[:, :5])
    Y = np.array(df_train.iloc[:, -1])
    Y[Y == 1] = 0
    Y[Y == 2] = 1
    return X, Y

def gk(x,y,w):
    h = np.exp(w.T@x)
    g = -(y-h/(1+h))*x
    return g

def logistic(X,Y):
    w = Weight()
    k, sigma, alpha, MaxN = 0, 10 ** (-5), 0.1, 8000
    for i in range(MaxN):
        j = i % len(X)
        x = X[j].reshape(-1,1)
        y = Y[j]
        g = gk(x,y,w)
        w = w - alpha*g
        k += 1
        if np.linalg.norm(g) < sigma:
            break
    return w

def Accuracy(W_matrix,df_test):
    X = np.array(df_test.iloc[:, :5])
    Y = np.array(df_test.iloc[:, -1])
    accur = []
    for x in X:
        lst = []
        for w in range(len(W_matrix)):
            total = W_matrix[w].T@x
            lst.append(total)
        if lst[0][0] < 0:
            accur.append(0)
        elif lst[0][0] > 0:
            if lst[1][0] < 0:
                accur.append(1)
            elif lst[1][0] > 0:
                accur.append(2)
    accuracy = sum(1 for x, y in zip(Y, accur) if x == y) / len(Y)
    return accuracy

if __name__ == '__main__':
    df_train,df_test,n,cate = load()
    W_matrix = []
    for i in range(len(cate)-1):
        pro = 'data_spilt{}'.format(i+1)
        X, Y = eval(pro)(df_train)
        w = logistic(X,Y)
        W_matrix.append(w)
    print('w:', W_matrix)
    accuracy = Accuracy(W_matrix,df_test)
    print('accuracy score:',accuracy)