Python机器学习（四）：logistic回归

logistic回归

logistic回归虽名为回归但其实做的是分类问题，是一个典型的线性分类器。

如上图中所示：将一组数据特征X输入分类器，它会输出一个预测值y帽（也可以表示为a）。
logistic回归的模型参数为W和b，其中W为一（n，1）矩阵，n为数据特征的维数，也就是X的维数，图中n=3。
由于$W^TX+b$ 是一个处于正负无穷间的实数，由于是二分类问题，所以我们的输出（输出定义为预测为1类别的概率）的其实是一个概率要在[0,1]内。所以要引入sigmoid函数，将正负无穷之间的输出转换到[0,1]内。

损失函数L(a，y)用来衡量模型在单个样本上的表现（(i)表示第i个样本）。

$$L(a^{(i)}，y^{(i)})= -[y^{(i)}loga^{(i)}+(1-y^{(i)})log(1-a^{(i)})]$$
成本函数J(W,b)用来衡量模型在全体训练集上的表现（m为训练集样本个数）：
$$J(W,b)=\frac{1}{m}\sum_{i=1}^m L(a^{(i)}，y^{(i)})=- \frac{1}{m}\sum_{i=1}^m [y^{(i)}loga^{(i)}+(1-y^{(i)})log(1-a^{(i)})]$$

logistic回归中的梯度下降法

在反向传播中：

$$\frac{dL}{da}=\frac{1-y}{1-a}-\frac{y}{a}$$
$$\frac{dL}{dz}=\frac{dL}{da}\frac{da}{dz}=(\frac{1-y}{1-a}-\frac{y}{a})(a(1-a))=a-y$$
当有m个训练样本时设：
$$d_a= \frac{dL}{da^{(i)}}$$
$$d_z=(a^{(i)}-y^{(i)})$$
$$d_w=\frac{dJ}{dW}=\frac{1}{m}XdZ^T$$
$dZ$为 $d_z$的矩阵表示（大写字母表示矩阵）,上述dZ和X矩阵相乘是自带求和功能
$$dZ=A-Y$$
$$d_b=\frac{dJ}{db}=\frac{1}{m}\sum_{i=1}^md_z$$
最终用 $d_w和d_b$配合学习率 $\alpha$来更新模型参数W和b：
$$W-=\alpha d_w$$
$$b-=\alpha d_b$$

代码实现

# -*- coding: utf-8 -*-
"""
Created on Wed Sep 27 20:12:48 2017

@author: YangYuan
"""

import numpy as np

def sigmoid(inX):
    return 1.0/(1+np.exp(-inX))

def plotBestFit(data,label,weight):
    import matplotlib.pyplot as plt
    x1 = []
    y1 = []
    x0 = []
    y0 = []
    for index,i in enumerate(label[0]):
        if i == 1:
            x1.append(data[1,index])
            y1.append(data[2,index])
        else:
            x0.append(data[1,index])
            y0.append(data[2,index])
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.scatter(x0,y0,s=30,c='red',marker='s')
    ax.scatter(x1,y1,s=30,c='green')

    x = np.arange(-4.0,4.0,0.1)
    y = -(x*weight[1][0]+weight[0][0])/weight[2][0]
    ax.plot(x,y)
    plt.xlabel('X1')
    plt.ylabel('X2')
    plt.show()

def main():  
    data_raw = np.loadtxt('testSet.txt')
    #分离数据与标签，并将标签转换为整形
    data = data_raw[:,:2].T
    label = data_raw[:,2:3].T.astype(int) 
    m,n = np.shape(data) #m,n = data.shape 

    temp = np.ones((1,n))
    data = np.insert(data,0,values=temp,axis=0)

    m,n = np.shape(data)
    w = np.random.randn(m,1)*0.01

    plotBestFit(data,label,w)

    for i in range(1000):
        dz = sigmoid(np.dot(w.T,data))-label
        dw = np.dot(data,dz.T)/n
        alpha = 0.1
        w -= alpha*dw


    plotBestFit(data,label,w)

if __name__ == '__main__':
    main()

testSet.txt中的内容

-0.017612   14.053064   0
-1.395634   4.662541    1
-0.752157   6.538620    0
-1.322371   7.152853    0
0.423363    11.054677   0
0.406704    7.067335    1
0.667394    12.741452   0
-2.460150   6.866805    1
0.569411    9.548755    0
-0.026632   10.427743   0
0.850433    6.920334    1
1.347183    13.175500   0
1.176813    3.167020    1
-1.781871   9.097953    0
-0.566606   5.749003    1
0.931635    1.589505    1
-0.024205   6.151823    1
-0.036453   2.690988    1
-0.196949   0.444165    1
1.014459    5.754399    1
1.985298    3.230619    1
-1.693453   -0.557540   1
-0.576525   11.778922   0
-0.346811   -1.678730   1
-2.124484   2.672471    1
1.217916    9.597015    0
-0.733928   9.098687    0
-3.642001   -1.618087   1
0.315985    3.523953    1
1.416614    9.619232    0
-0.386323   3.989286    1
0.556921    8.294984    1
1.224863    11.587360   0
-1.347803   -2.406051   1
1.196604    4.951851    1
0.275221    9.543647    0
0.470575    9.332488    0
-1.889567   9.542662    0
-1.527893   12.150579   0
-1.185247   11.309318   0
-0.445678   3.297303    1
1.042222    6.105155    1
-0.618787   10.320986   0
1.152083    0.548467    1
0.828534    2.676045    1
-1.237728   10.549033   0
-0.683565   -2.166125   1
0.229456    5.921938    1
-0.959885   11.555336   0
0.492911    10.993324   0
0.184992    8.721488    0
-0.355715   10.325976   0
-0.397822   8.058397    0
0.824839    13.730343   0
1.507278    5.027866    1
0.099671    6.835839    1
-0.344008   10.717485   0
1.785928    7.718645    1
-0.918801   11.560217   0
-0.364009   4.747300    1
-0.841722   4.119083    1
0.490426    1.960539    1
-0.007194   9.075792    0
0.356107    12.447863   0
0.342578    12.281162   0
-0.810823   -1.466018   1
2.530777    6.476801    1
1.296683    11.607559   0
0.475487    12.040035   0
-0.783277   11.009725   0
0.074798    11.023650   0
-1.337472   0.468339    1
-0.102781   13.763651   0
-0.147324   2.874846    1
0.518389    9.887035    0
1.015399    7.571882    0
-1.658086   -0.027255   1
1.319944    2.171228    1
2.056216    5.019981    1
-0.851633   4.375691    1
-1.510047   6.061992    0
-1.076637   -3.181888   1
1.821096    10.283990   0
3.010150    8.401766    1
-1.099458   1.688274    1
-0.834872   -1.733869   1
-0.846637   3.849075    1
1.400102    12.628781   0
1.752842    5.468166    1
0.078557    0.059736    1
0.089392    -0.715300   1
1.825662    12.693808   0
0.197445    9.744638    0
0.126117    0.922311    1
-0.679797   1.220530    1
0.677983    2.556666    1
0.761349    10.693862   0
-2.168791   0.143632    1
1.388610    9.341997    0
0.317029    14.739025   0