机器学习笔记4-Logistic回归

最新推荐文章于 2021-11-22 19:00:30 发布

Tobesix

最新推荐文章于 2021-11-22 19:00:30 发布

阅读量186

点赞数

分类专栏：机器学习篇文章标签：机器学习

本文链接：https://blog.csdn.net/intricate_/article/details/105015696

版权

机器学习篇专栏收录该内容

21 篇文章 7 订阅

订阅专栏

逻辑回归(分类算法)

Logistic回归为最简单的分类算法之一，可以使用批量/随机梯度下降算法来优化Loss。

Sigmoid函数：

${\hat{y}}=\sigma ({z})=\frac{1}{1+{e}^{-{z}}}=\frac{1}{1+{e}^{-(\hat{\omega}^{T}{x})}}$ $\frac{\partial (\hat{y})}{\partial(z)}=\hat{y}*(1-\hat{y})$

损失函数：交叉熵

这里为什么不用平方差损失函数而用交叉熵呢？

平方差损失函数：

$L(\theta )=\frac{1}{2}\sum_{i=1}^{n}(y_{i}-\hat{y}^{(i)})^{2}$

$\hat{y}^{(i)}=\sigma (w^{T}x+b)$

求导后得到的梯度为 $(\hat{y}^{(i)}-y_{i})\sigma {}'({y}^{(i)})x$ ，和激活函数的梯度成正比，会出现梯度弥散，梯度下降较慢。

而交叉熵损失函数：

$L(\theta )=-\frac{1}{m}\sum_{i=1}^{m}({y}^{(i)}\log(\hat{y}^{(i)})+(1-{y}^{(i)})\log (1-\hat{y}^{(i)}))$

求导后得到的梯度为 $({x}^{(i)}({\hat{y}}^{(i)}-{y}^{(i)}))$ ，和激活函数无关。

交叉熵损失函数简化后：

$L(\theta )=-\frac{1}{m}\sum_{i=1}^{m}({y}^{(i)}\log(\frac{1}{1+{e}^{-{z}^{i}}})+(1-{y}^{(i)})\log (1-\frac{1}{1+{e}^{-{z}^{i}}}))$

$=\frac{1}{m}\sum_{i=1}^{m}({y}^{(i)}\log(1+{e}^{-{z}^{i}})+(1-{y}^{(i)})(-\log \frac{{e}^{-{z}^{i}}}{1+{e}^{-{z}^{i}}}))$

$=\frac{1}{m}\sum_{i=1}^{m}({y}^{(i)}\log(1+{e}^{-{z}^{i}})+(1-{y}^{(i)})(\log (1+{e}^{-{z}^{i}})+{z}^{i}))$

$=\frac{1}{m}\sum_{i=1}^{m}(\log(1+{e}^{-{z}^{i}})+{z}^{i}-{y}^{(i)}{z}^{i})$

$=\frac{1}{m}\sum_{i=1}^{m}(\log(1+{e}^{{z}^{i}})-{y}^{(i)}{z}^{i})$

求导过程：

$\frac{\partial }{\partial \omega }=\frac{1}{m}\sum_{i=1}^{m}(\frac{1}{1+{e}^{{\omega}^{T}{x}^{(i)}}}*{e}^{{\omega}^{T}{x}^{(i)}}*{x}^{(i)}-{y}^{(i)}{x}^{(i)})$

$=\frac{1}{m}\sum_{i=1}^{m}({x}^{(i)}(\frac{1}{1+{e}^{-{\omega}^{T}{x}^{(i)}}}-{y}^{(i)}))$

$=\frac{1}{m}\sum_{i=1}^{m}({x}^{(i)}({\hat{y}}^{(i)}-{y}^{(i)}))$

Python代码如下：

import numpy as np

# sigmoid函数
def sigmoid(inX):
    if inX >= 0:
        return 1.0 / (1 + np.exp(-inX))
    else:
        return np.exp(inX) / (1 + np.exp(inX))

# 批量梯度下降
def plGradAscent(dataMatrix, classLabels,numIter=150):
    m, n = dataMatrix.shape
    weights = np.ones(n)
    for j in range(numIter):  # j为迭代次数
        alpha = 4 / (1.0 + j) + 0.0001  # alpha随迭代次数每次减少
        error = 0.0
        for i in range(m):
            h = sigmoid(sum(dataMatrix[i] * weights))
            suberror = h - classLabels[i]
            error += suberror
        weights = weights - alpha * error * (1/m)
    return weights

# 随机梯度下降
def stocGradAscent(dataMatrix, classLabels, numIter=150):
    m, n = dataMatrix.shape
    weights = np.ones(n)
    for j in range(numIter):  # j为迭代次数
        dataIndex = list(range(m))
        for i in range(m):
            alpha = 4 / (1.0 + j + i) + 0.0001  # alpha随迭代次数每次减少
            randIndex = int(np.random.uniform(0, len(dataIndex)))  # 随机选取样本来更新权值系数，较少周期性波动
            h = sigmoid(sum(dataMatrix[dataIndex[randIndex]] * weights))
            error = h - classLabels[dataIndex[randIndex]]
            weights = weights - alpha * error * dataMatrix[dataIndex[randIndex]]
            del (dataIndex[randIndex])  # 删除选择过的该样本
    return weights

# 得到测试结果
def classifyVector(inX, weights):
    prob = sigmoid(sum(inX*weights))
    if prob > 0.5:
        return 1.0
    else:
        return 0.0

# 测试
def colicTest():
    frTrain = open('horseColicTraining.txt')
    frTest = open('horseColicTest.txt')
    trainingData = []
    trainingLabels = []
    for line in frTrain.readlines():
        currLine = [float(example) for example in line.strip().split('\t')]
        trainingData.append(currLine[:-1])
        trainingLabels.append(currLine[-1])
    trainWeights = stocGradAscent(np.array(trainingData), trainingLabels, 500)
    errorCount = 0
    numTestVec = 0.0
    for line in frTest.readlines():
        numTestVec += 1.0
        currLine = [float(example) for example in line.strip().split('\t')]
        lineArr = currLine[:-1]
        if int(classifyVector(np.array(lineArr), trainWeights)) != int(currLine[-1]):
            errorCount += 1
    errorRate = (float(errorCount) / numTestVec)
    print("the error rate of this test is: %f" % errorRate)
    return errorRate

if __name__ == '__main__':
    colicTest()