逻辑回归

Logistic Regression

推导:

logit->sigmoid=P

$$\begin{array}{rll} \text{记一个事件发生的概率为:}&P(y=1|x),\text{简记为P;不发生则为1-P} \\ \\\text{logit函数:} \\ &\log it(x)=ln(\frac{P}{1-P})=w_0+w_1x_1+w_2 x_2+ \cdots+w_mx_m \\ \\\text{推导出:}&P(y=1|x)=P=\frac{1}{1+e^{-z}} \\ &\quad \text{ 其中:}z=w_0+w_1 x_1+w_2 x_2+ \cdots +w_m x_m \\ \\ \text{S形函数:}& \\ &f(z)=\frac{1}{1+e^{-z}} \\ \\&\text{定义域}(-\infty,+\infty),\text{值域(0,1)} \end{array}$$

cost 函数

似然函数->max;-ln(似然函数)->min(v凸函数)

$$\begin{array}{rclllll} \\\text{似然函数:} \\&L=\prod_i P_i \\\text{} \\&L(w)&=\prod_i^N P(x_i|y=1)^{y_{i}} \cdot P(x_i|y=0)^{1-y_{i}} \\ \\&似然函数最大化& \mathop{arg}_w \ max \ \ L(w) \\&\to\text{等价于}&\mathop{arg}_w \ min \ -L(w) \\ \\\text{二分类中:}&P(y=0)&=1-P(y=1) \\&\text{带入}L(w)&\text{并取自然对数}ln\text{,并取负,可得Loss损失函数:} \\ \text{}& \\ & cost(w)&=-y_{i}\sum_i ln P-(1-y_{i})\sum_i ln(1-P) \\\text{整理,得:} \\&&=-\sum_i^n y_{i} z+\sum_i^n ln(1+e^z) \\ \\&&=-\sum_i^n y_{i}w\cdot x_i+\sum_i^n ln(1+e^{w\cdot x_i}) \\ \\&\text{求解:}&\mathop{arg}_w \ min \ cost(w) \\&\text{}&\text{在区间}(0,1)-ln x;-ln(1-x)\text{ 均为v凸函数,局部min=全局min} \end{array}$$

---

优化求解

$$\begin{array}{rll} \\\text{对 } L(w)\text{ 求 }w&\text{的 偏 导 :} \\&ln(1+e^{w\cdot x_i})' &=\frac{(1+e^{w\cdot x_i})'}{(1+e^{w\cdot x_i})} =\frac{e^{w\cdot x_i}\ (w\cdot x_i)'}{(1+e^{w\cdot x_i})} =\frac{e^{w\cdot x_i}\ x_i}{(1+e^{w\cdot x_i})} \\ \\&&=\frac{1}{1+e^{-w\cdot x_i}}\ x_i \\ \\&&=f(w\cdot x_i)\ x_i=P_i\ x_i \\ \\ & \frac{\partial}{\partial w} L(w)& =-\sum_i^n y_{i}\ x_i+\sum_i^n [ln(1+e^{w\cdot x_i})]' \\ \\&&=\sum_i^n(P_i-y_i)\ x_i \\ \end{array}$$

梯度法求解:BGD,SGD

• BGD-> python code […]

其他方法:共轭梯度法、BFGS、L-BFGS

• 见 … .待编辑

 
 

 

python代码实现 BGD SGD

import numpy as np

def sigmoid(z):
    return 1/(1+np.exp(-z))

def gradDesc(x,y,lr=0.01,loop=402,opt='gd'):
    w=np.zeros(len(x[0]))
    b=0
    if opt'gd': #梯度下降法
        for i in range(loop):
            error=sigmoid((x.dot(w)+b))-y #P_i-y_i
            #grad_w=\sum_i^N (p_i-y_i)*x_i
            w-=lr*np.dot(error,x)
            b-=lr*(error.sum())   #grad_b,(x_i=1)即可
    elif opt'sgd': #随机梯度下降法
        for looper in range(loop):
            for i in range(len(y)):
                #  print(w.dot(x[i],w))
                error=sigmoid(w.dot(x[i])+b)-(y[i])
                w-=lr*error*(x[i])
                b-=lr*error
    elif opt'xx':
        pass
    else:
        print('Not support this optimize method')
        #raise error
    return w,b   


if __name__ '__main__':
    # x_train,y_train
    x=np.array([[0,0],[1,1],[0,1],[1,0],[2,1],[3,3],[1,2],[1,3],[0.9,0.9]])
    y=np.array([0,0,0,0,1,1,1,1,1])
    w,b=gradDesc(x,y,lr=0.001,opt='sgd',loop=10000)
    print(':',w/b,b)
    import matplotlib.pyplot as plt
    plt.scatter(x[:,0],x[:,1],c=y)
    x_plot=np.array([x.min(),x.max()],dtype='float32')
    y_plot=-(x_plot*w[0]+b)/(w[1])
    plt.plot(x_plot,y_plot)
    plt.show()