线性回归分析——含python代码

  假设样本空间为$d$个维度,用$\boldsymbol{x}$={$x_1,x_2,x_3,...,x_d$}来表示一个样本点，线性回归的目标是用$d+1$个参数来拟合样本点的输入和输出。通常我们会将$\boldsymbol{x}$扩充为$d+1$维的向量$\boldsymbol{x}$={$x_0,x_1,x_2,x_3,...,x_d$}，第$x_0$设为1作为偏置项。线性回归表达式如下：

$$f=\sum_{i=0}^{d+1} \theta_i x_i$$
用向量可以表示为：
$$f(x)=\boldsymbol{\theta}^T \boldsymbol{x}$$
将表达式代入均方误差函数 $(MSE)$中:
$$J = \frac{1}{N}\sum_{i=1}^{N}(y_i-f(\boldsymbol{x}_i))^2$$
令全部样本表示为： $X$={$\boldsymbol{x}_1,\boldsymbol{x}_2,\boldsymbol{x}_3,...,\boldsymbol{x}_N$} ^$T$，对应输出表示为$Y$={$y_1,y_2,y_3,...,y_N$}^$T$，损失函数可以简化为：
$$J =\frac{1}{N}||X\boldsymbol{\theta}-Y||^2$$
我们的目标是让损失函数最小，令其对$\boldsymbol{\theta}$的偏导数为0,则有：
$$\begin{eqnarray} \frac{\partial J}{\partial \boldsymbol{\theta}}&=&\frac{\partial}{\partial \boldsymbol{\theta}}[\frac{1}{N}(\boldsymbol{\theta}^TX^TX\boldsymbol{\theta}-2\boldsymbol{\theta}^TX^TY+Y^TY)] \\ &=&\frac{1}{N}(2X^TX\boldsymbol{\theta}-2X^TY)\\ &=&0 \end{eqnarray}$$
解得：$\boldsymbol{\theta}=(X^TX)^{-1}X^TY$。
  显然，$X^TX$要是可逆的，通常情况下都能满足因为$N>>d+1$。事实上$X^TX$也存在不可逆的情况，这种情况下我们可以选择求伪逆或者梯度下降法来解决问题。
  如果要利用线性回归来解决二分类问题，$y_i$就不再是连续值，而是1(正样本)或者0/-1(负样本)。同样的求出权重向量后对测试样本进行预测，可以用0.5/0作为阈值来划分正负样本。

代码块

理解了线性回归和梯度下降的基本原理，用python撸出来也就10分钟的时间：

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import scale
from random import random
from sklearn.model_selection import train_test_split

class LinearRegression(object):
    weight = np.array([])
    way = 'gd'
    def __init__(self, training_way = 'gd'):
        self.way = training_way
    def gradientDescent(self, X, Y, alpha, epoch):
        W = np.random.normal(0,1,size=(X.shape[1],))
        for i in range(epoch):
            W -= alpha*(X.T).dot(X.dot(W)-Y)/X.shape[0]
        return W

    def fit(self, train_data, train_target, alpha = 0.1, epoch = 300):
        X = np.ones((train_data.shape[0], train_data.shape[1]+1))
        X[:,0:-1] = train_data
        Y = train_target
        if self.way == 'gd':
            self.weight = self.gradientDescent(X, Y, alpha, epoch)
        else:
            self.weight = np.linalg.inv((X.T).dot(X)).dot(X.T).dot(Y)

    def predict(self, test_data):
        X = np.ones((test_data.shape[0], test_data.shape[1]+1))
        X[:,0:-1] = test_data
        return X.dot(self.weight)

    def evaluate(self, predict_target, test_target):
        predict_target[predict_target>=0.5] = 1
        predict_target[predict_target<0.5] = 0
        return sum(predict_target==test_target)/len(predict_target)

if __name__ == "__main__":
    cancer = load_breast_cancer()
    xtr, xval, ytr, yval = train_test_split(cancer.data, cancer.target, \
    test_size=0.2, random_state=7)
    linear = LinearRegression(training_way = 'gd')
    linear.fit(xtr, ytr, alpha = 0.05, epoch = 1000)
    predict = linear.predict(xval)
    print('linear regression accruacy:',linear.evaluate(predict, yval))