最小二乘法推导及实现

一、公式推导

直线：
$$y=wx+b$$
损失函数：
$$L=\sum _{i=1}^n(w{x}_i+b-y_i{)}^2$$
最小二乘法的核心思想就是让损失函数最小，即找出损失函数的最小值。

在高等数学中，要求一个函数的最小值的思路为：1.对函数求导 2.让导数等于0

求某一个w,b使得L最小

1. 先对L求偏导
   $$\frac{\partial L}{\partial b}=\frac{\partial {\sum }_{i=1}^n(w{x}_i+b-y_i{)}^2}{\partial b}$$

   $$\frac{\partial L}{\partial b}=2\sum _{i=1}^n(w{x}_i+b-y_i)$$

   $$\frac{\partial L}{\partial b}=2(\sum _{i=1}^{n}w{x}_i+\sum _{i=1}^{n}b-\sum _{i=1}^n{y}_i)$$

   设
   $$\bar{x}=\frac{{\sum }_{i=1}^n{x}_i}{n}$$

   $$\frac{\partial L}{\partial b}=2(wn\bar{x}+nb-n\bar{y})$$

   $$\frac{\partial L}{\partial b}=2n(w\bar{x}+b-\bar{y})$$

   要使
   $$\frac{\partial L}{\partial b}=0$$
   则
   $$b=\bar{y}-w\bar{x}$$
   2.L对w求偏导
   $$\frac{\partial L}{\partial w}=\sum _{i=1}^{n}2(w{x}_i+b-y_i)x_i$$

$$\frac{\partial L}{\partial w}=2\sum _{i=1}^n(w{x}_i^2+b{x}_i-x_i{y}_i)$$

$$\frac{\partial L}{\partial w}=2(\sum _{i=1}^{n}w{x}_i^2+\sum _{i=1}^{n}b{x}_i-\sum _{i=1}^n{x}_i{y}_i)$$

由于
$$b=\bar{y}-w\bar{x}$$
所以
$$\frac{\partial L}{\partial w}=2(\sum _{i=1}^{n}w{x}_i^2+\sum _{i=1}^n(\bar{y}-w\bar{x})x_i-\sum _{i=1}^n{x}_i{y}_i)$$

$$\frac{\partial L}{\partial w}=2(\sum _{i=1}^{n}w{x}_i^2+n\bar{x}(\bar{y}-w\bar{x})-\sum _{i=1}^n{x}_i{y}_i)$$

$$\frac{\partial L}{\partial w}=2(\sum _{i=1}^{n}w{x}_i^2+n\bar{x}\bar{y}-nw{\bar{x}}^2-\sum _{i=1}^n{x}_i{y}_i)$$

$$\frac{\partial L}{\partial w}=2(w(\sum _{i=1}^n{x}_i^2-n{\bar{x}}^2)+n\bar{x}\bar{y}-\sum _{i=1}^n{x}_i{y}_i))$$

要使
$$\frac{\partial L}{\partial w}=0$$
即
$$w=\frac{{\sum }_{i=1}^n{x}_i{y}_i-n\bar{x}\bar{y}}{{\sum }_{i=1}^n{x}_i^2-n{\bar{x}}^2}$$

$$w=\frac{{\sum }_{i=1}^n(x-\bar{x})(y-\bar{y})}{{\sum }_{i=1}^n(x-\bar{x}{)}^2}$$

结论：
$$w=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$$

$$b=\bar{y}-w\bar{x}$$

二、代码实现

import numpy as np
x = np.linspace(0, 100, 200)
noise = np.random.normal(loc=10, scale=20, size=200)
y = 3*x + 10 + noise

import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif'] = ['SimHei']
plt.rcParams['axes.unicode_minus'] = False

plt.scatter(x, y, c='c')
plt.title('X-Y数据图')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

代码实现

# 先写一个计算loss的函数
def cal_L(w, b, x, y):
    L = 0
    for i in range(len(x)):
        L += (w*x[i] + b - y[i])**2
    return L   

def find_wb(x, y):
    x_mean = np.mean(x)
    y_mean = np.mean(y)
    w = sum((x - x_mean)*(y-y_mean)) / sum(((x - x_mean)**2))
    b = y_mean - w*x_mean
    return w,b



w, b = find_wb(x, y)

cal_L(w, b, x, y)

plt.scatter(x, y, c='c')
y_pre = w*x+b
plt.plot(x, y_pre, 'r-')