Logistic回归推导（二）--梯度上升法及纯python实现

最新推荐文章于 2022-07-13 21:00:03 发布

气泡水、

最新推荐文章于 2022-07-13 21:00:03 发布

阅读量636

点赞数 3

分类专栏：机器学习文章标签： python 机器学习逻辑回归

本文链接：https://blog.csdn.net/m0_50117360/article/details/108969132

版权

机器学习专栏收录该内容

7 篇文章 3 订阅

订阅专栏

1、假设函数

假设函数为：
$\begin{aligned} P(y|x;\theta)&=[h_{\theta}(x)]^{y}[1-h_{\theta}(x)]^{(1-y)}\\ &=(\frac{1}{1+e^{-\theta^{T}x}})^{y}(1-\frac{1}{1+e^{-\theta^{T}x}})^{1-y} \end{aligned}$

2、最大似然估计

似然函数
$\begin{aligned} L(\theta)&=\prod_{i=1}^{N}P(y^{(i)}|x^{i};\theta)\\ &=\prod_{i=1}^{N}[h_{\theta}(x^{(i)})]^{y^{(i)}}[1-h_{\theta}(x^{(i)})]^{(1-y^{(i)})}\\ &=\prod_{i=1}^{N}(\frac{1}{1+e^{-\theta^{T}x^{(i)}}})^{y^{(i)}}(1-\frac{1}{1+e^{-\theta^{T}x^{(i)}}})^{1-y^{(i)}} \end{aligned}$
Logistic回归的目标函数即为对数似然函数，进行最大似然估计则可求解
$\underset{\theta}{max}L(\theta)\Leftrightarrow \underset{\theta}{max}\sum_{i=1}^{n}[y^{(i)}lnh_{\theta}(x^{(i)})+(1-y^{(i)})ln(1-h_{\theta}(x^{(i)}))]$

3、梯度上升法求解

梯度
$\begin{aligned} \frac{dL(\theta)}{d\theta}&=\frac{d}{d\theta}\sum_{i=1}^{N}[y^{(i)}lnh_{\theta}(x^{(i)})+(1-y^{(i)})ln(1-h_{\theta}(x^{(i)}))]\\ &=\sum_{i=1}^{N}[y^{(i)} \cdot \frac{1}{h_{\theta}(x^{(i)})}-(1-y^{(i)})\cdot \frac{1}{1-h_{\theta}(x^{(i)})}] \cdot \frac{\partial}{\partial\theta}h_{\theta}(x^{(i)})\\ &=\sum_{i=1}^{N}[y^{(i)} \cdot \frac{1}{h_{\theta}(x^{(i)})}-(1-y^{(i)})\cdot \frac{1}{1-h_{\theta}(x^{(i)})}] \cdot h_{\theta}(x^{(i)})\cdot(1-h_{\theta}(x^{(i)})) \cdot \frac{\partial}{\partial\theta}\theta^{T}x^{(i)}\\ &=\sum_{i=1}^{N}[y^{(i)} \cdot (1-h_{\theta}(x^{(i)})) -(1-y^{(i)})\cdot h_{\theta}(x^{(i)})] x^{(i)}\\ &=\sum_{i=1}^{N}(y^{(i)}-h_{\theta}(x^{(i)}))x^{(i)} \quad \quad {\color{red}误差✖特征} \end{aligned}$
梯度上升优化：
$\begin{aligned} \theta:=\theta+\alpha \frac{dL(\theta)}{d\theta}=\theta+\alpha\sum_{i=1}^{N}(y^{(i)}-h_{\theta}(x^{(i)}))x^{(i)} \end{aligned}$

4、纯python实现

代码入下

import numpy as np
import matplotlib.pyplot as plt
import time


# 加载数据
def load_data():
    X_train = np.loadtxt("./Exam/train/x.txt")
    Y_train = np.loadtxt("./Exam/train/y.txt", dtype=int)
    X_test = np.loadtxt("./Exam/test/x.txt")
    Y_test = np.loadtxt("./Exam/test/y.txt", dtype=int)
    return X_train, Y_train, X_test, Y_test


# Logistic回归类
class Logistic(object):
    def __init__(self, X_train, Y_train):
        self.X_train = X_train
        self.Y_train = Y_train
        # M:特征数，N：样本数
        self.M = X_train.shape[1]
        self.N = X_train.shape[0]
        self.lr = 0.005
        self.train()

    def normalization(self):
        # 均值方差归一化
        mean = np.mean(self.X_train)
        variance = np.std(self.X_train)
        self.X_train = (self.X_train - mean)/variance
        self.X_train = np.insert(self.X_train, 0, values=1.0, axis=1)
        self.Y_train = self.Y_train.reshape(self.N, 1)
        self.M += 1
        
    def sigmoid(self, X):
        eta = -np.dot(X, self.theta)  # N*1
        H = np.exp(eta)
        H = 1.0 / (1.0 + H)
        return H

    def Gradient_ascent(self):
        self.theta = -np.ones((self.M, 1))
        for i in range(500):
            self.H = self.sigmoid(self.X_train)
            self.theta += self.lr * np.dot(self.X_train.T, (self.Y_train - self.H))
            self.loss = -np.sum(self.Y_train * np.log(self.H) + (1.0 - self.Y_train) * np.log(1 - self.H))/self.N
            print("iter: %d, loss: %f" % (i, self.loss))
        print(self.theta)

    def train(self):
        self.normalization()
        self.Gradient_ascent()


if __name__ == "__main__":
    X_train, Y_train, X_test, Y_test = load_data()
    Logistic(X_train, Y_train)