机器学习实战-Logistic回归

首先，Logistics回归的损失函数不是和线性回归的损失函数一样，并非平方误差函数，因为平方误差用在Logistic中，代价函数是一个非凸函数，会有多个局部最小值。
推导过程如下：
假设函数：
$$h_{\theta }(x)=g({\theta }^{T}x)=\frac{1}{1+e^{-{\theta }^{T}x}}$$
sigmoid函数及其导数：
$$g(x)=\frac{1}{1+e^{-x}},g^{{}^{\prime }}(x)=g(x)[1-g(x)]$$
概率分布：
$$\{\begin{array}{ll}& \text{P}(y=1|x;\theta )=h_{\theta }(x)\\ & \text{P}(y=0|x;\theta )=1-h_{\theta }(x)\end{array}$$
合并为：
$$P(y|x;\theta )=[h_{\theta }(x){]}^y[1-h_{\theta }(x){]}^{1-y}$$
采用最大似然估计进行参数估计，似然函数：
$$L(\theta )=\prod _{i=1}^{m}P(y^{(i)}|x^{(i)};\theta )$$
$$=\prod _{i=1}^m[h_{\theta }(x^{(i)}){]}^{y^{(i)}}[1-h_{\theta }(x^{(i)}){]}^{1-y^{(i)}}$$
似然函数取对数：
$$l(\theta )=logL(\theta )=\sum _{i=1}^m{y}^{(i)}log{h}_{\theta }(x^{(i)})+(1-y^{(i)})(1-log{h}_{\theta }(x^{(i)}))$$
所以，我们要使似然函数取最大值，采用梯度上升的方法求$l(\theta )$的最大值。
在吴恩达机器学习视频中，他令损失函数：
$$J(\theta )=-\frac{1}{m}l(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}log{h}_{\theta }(x^{(i)})+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))$$
所以求似然函数$l(\theta )$的最大值，相当于求损失函数$J(\theta )$的最小值，采用梯度下降。
所以，在吴恩达机器学习视频中，他直接给出的logistics的误差函数就是这样的：
$$Cost(h_{\theta }(x),y)=\{\begin{array}{ll}& \text{-}log{h}_{\theta }(x),当y=1\\ & \text{-}log(1-h_{\theta }(x)),当y=0\end{array}=-y^{(i)}log{h}_{\theta }(x^{(i)})-(1-y^{(i)})(1-log{h}_{\theta }(x^{(i)}))$$
梯度求解过程：
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}log{h}_{\theta }(x^{(i)})+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))$$
$$\frac{\partial }{\partial {\theta }_j}J(\theta )=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}\frac{\partial }{\partial {\theta }_j}{h}_{\theta }(x^{(i)})+(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})}\frac{\partial }{\partial {\theta }_j}(1-h_{\theta }(x^{(i)}))]$$
$$=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}\frac{\partial }{\partial {\theta }_j}{h}_{\theta }(x^{(i)})-(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})}\frac{\partial }{\partial {\theta }_j}{h}_{\theta }(x^{(i)})]$$
$$=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}-(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})}]\frac{\partial }{\partial {\theta }_j}{h}_{\theta }(x^{(i)})$$
$$=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}\frac{1}{h_{\theta }(x^{(i)})}-(1-y^{(i)})\frac{1}{1-h_{\theta }(x^{(i)})}]h_{\theta }(x^{(i)})[1-h_{\theta }(x^{(i)})]x_j^{(i)}$$
$$=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}(1-h_{\theta }(x^{(i)}))-(1-y^{(i)})h_{\theta }(x^{(i)})]x_j^{(i)}$$
$$=-\frac{1}{m}\sum _{i=1}^m[y^{(i)}-h_{\theta }(x^{(i)})]x_j^{(i)}$$
梯度更新过程：
$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )={\theta }_j-\alpha (-\frac{1}{m}\sum _{i=1}^m[y^{(i)}-h_{\theta }(x^{(i)})]x_j^{(i)})$$
$$={\theta }_j+\alpha (\frac{1}{m}\sum _{i=1}^m[y^{(i)}-h_{\theta }(x^{(i)})]x_j^{(i)})$$

1.数据处理

testSet.txt：

-0.017612	14.053064	0
-1.395634	4.662541	1
-0.752157	6.538620	0
-1.322371	7.152853	0
0.423363	11.054677	0
0.406704	7.067335	1
0.667394	12.741452	0
-2.460150	6.866805	1
0.569411	9.548755	0
-0.026632	10.427743	0
0.850433	6.920334	1
1.347183	13.175500	0
1.176813	3.167020	1
-1.781871	9.097953	0
-0.566606	5.749003	1
0.931635	1.589505	1
-0.024205	6.151823	1
-0.036453	2.690988	1
-0.196949	0.444165	1
1.014459	5.754399	1
1.985298	3.230619	1
-1.693453	-0.557540	1
-0.576525	11.778922	0
-0.346811	-1.678730	1
-2.124484	2.672471	1
1.217916	9.597015	0
-0.733928	9.098687	0
-3.642001	-1.618087	1
0.315985	3.523953	1
1.416614	9.619232	0
-0.386323	3.989286	1
0.556921	8.294984	1
1.224863	11.587360	0
-1.347803	-2.406051	1
1.196604	4.951851	1
0.275221	9.543647	0
0.470575	9.332488	0
-1.889567	9.542662	0
-1.527893	12.150579	0
-1.185247	11.309318	0
-0.445678	3.297303	1
1.042222	6.105155	1
-0.618787	10.320986	0
1.152083	0.548467	1
0.828534	2.676045	1
-1.237728	10.549033	0
-0.683565	-2.166125	1
0.229456	5.921938	1
-0.959885	11.555336	0
0.492911	10.993324	0
0.184992	8.721488	0
-0.355715	10.325976	0
-0.397822	8.058397	0
0.824839	13.730343	0
1.507278	5.027866	1
0.099671	6.835839	1
-0.344008	10.717485	0
1.785928	7.718645	1
-0.918801	11.560217	0
-0.364009	4.747300	1
-0.841722	4.119083	1
0.490426	1.960539	1
-0.007194	9.075792	0
0.356107	12.447863	0
0.342578	12.281162	0
-0.810823	-1.466018	1
2.530777	6.476801	1
1.296683	11.607559	0
0.475487	12.040035	0
-0.783277	11.009725	0
0.074798	11.023650	0
-1.337472	0.468339	1
-0.102781	13.763651	0
-0.147324	2.874846	1
0.518389	9.887035	0
1.015399	7.571882	0
-1.658086	-0.027255	1
1.319944	2.171228	1
2.056216	5.019981	1
-0.851633	4.375691	1
-1.510047	6.061992	0
-1.076637	-3.181888	1
1.821096	10.283990	0
3.010150	8.401766	1
-1.099458	1.688274	1
-0.834872	-1.733869	1
-0.846637	3.849075	1
1.400102	12.628781	0
1.752842	5.468166	1
0.078557	0.059736	1
0.089392	-0.715300	1
1.825662	12.693808	0
0.197445	9.744638	0
0.126117	0.922311	1
-0.679797	1.220530	1
0.677983	2.556666	1
0.761349	10.693862	0
-2.168791	0.143632	1
1.388610	9.341997	0
0.317029	14.739025	0

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2，y=\frac{1}{1+e^{-h_{\theta (x)}}}$$

2.梯度下降

参考链接：https://blog.csdn.net/achuo/article/details/51160101

'''data'''
def loadDataSet():
    f=open('testSet.txt')
    data=[]
    label=[]
    for line in f.readlines():
        tmp=line.strip().split()
        data.append((1,float(tmp[0]),float(tmp[1])))
        label.append(int(tmp[2]))
    #data:(100,3)
    #label:(100)
    return data,label

def sigmoid(x):
    return 1.0/(1+np.exp(-x))

'''梯度下降'''
def gradAscent(data,label):
    #data:(100,3)
    #label:(100)
    
    #(100,3)
    dataM = np.mat(data)
    #(100,1)
    labelM = np.mat(label).transpose()
    
    #m个数据，n个特征（包括第0维特征1.0）
    m,n = np.shape(data)
    alpha = 0.001
    iters = 500
    #(3,1)
    weights = np.ones((n,1))
    for i in range(iters):
        #(100,3)*(3,1)=(100,1)
        h = sigmoid(dataM*weights)
        error = labelM-h #对应上面的梯度更新过程
        weights = weights + alpha*dataM.transpose()*error
    return weights

$$y=\frac{1}{1+e^{-{\mathbf{\theta }}^{\mathbf{T}}\mathbf{x}}}=\frac{1}{1+e^{{\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2}}$$
更新公式：
$${\theta }_j:={\theta }_j-\alpha \cdot \sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}$$