在统计学中，线性回归是利用线性回归方程的最小平方函数对自变量和因变量之间关系进行建模的一种回归分析。这种函数是称为回归系数的模型参数的线性组合。只有一个自变量的情况称为简单线性回归，多于一个自变量情况的叫做多元线性回归。
其表达式为
ϵ
(
i
)
=
y
(
i
)
−
θ
T
x
(
i
)
\epsilon^{(i)} = y^{(i)} - \theta^Tx^{(i)}
ϵ(i)=y(i)−θTx(i)
线性回归的假设前提是误差服从正态分布，即因变量服从正态分布。当 μ = 0，σ = 1 时的正态分布是标准正态分布。
f
(
x
)
=
1
2
π
σ
e
x
p
(
−
(
x
−
μ
)
2
2
σ
2
)
f(x) = \frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)
f(x)=2πσ1exp(−2σ2(x−μ)2)
由于误差服从高斯分布
p
(
ϵ
(
i
)
)
=
1
2
π
σ
e
x
p
(
−
(
ϵ
(
i
)
)
2
2
σ
2
)
p(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{(\epsilon^{(i)})^2}{2\sigma^2}\right)
p(ϵ(i))=2πσ1exp(−2σ2(ϵ(i))2)
即
p
(
y
(
i
)
∣
x
(
i
)
;
θ
)
=
1
2
π
σ
e
x
p
(
−
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
2
σ
2
)
p(y^{(i)} | x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}\right)
p(y(i)∣x(i);θ)=2πσ1exp(−2σ2(y(i)−θTx(i))2)
似然函数
L
(
θ
)
=
∏
i
=
1
n
p
(
y
(
i
)
∣
x
(
i
)
;
θ
)
=
∏
i
=
1
n
1
2
π
σ
e
x
p
(
−
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
2
σ
2
)
L(\theta) = \prod_{i=1}^{n}p(y^{(i)} | x^{(i)}; \theta) = \prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}\right)
L(θ)=i=1∏np(y(i)∣x(i);θ)=i=1∏n2πσ1exp(−2σ2(y(i)−θTx(i))2)
对数似然
l
o
g
L
(
θ
)
=
l
o
g
∏
i
=
1
n
1
2
π
σ
e
x
p
(
−
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
2
σ
2
)
=
∑
i
=
1
n
l
o
g
1
2
π
σ
e
x
p
(
−
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
2
σ
2
)
=
m
⋅
l
o
g
1
2
π
σ
−
1
2
⋅
1
σ
2
∑
i
=
1
n
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
\begin{aligned} logL(\theta) &= log\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp\left(-\frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}\right) \\ &= \sum_{i=1}^{n}log\frac{1}{\sqrt{2\pi}\sigma}exp\left(- \frac{(y^{(i)} - \theta^Tx^{(i)})^2}{2\sigma^2}\right) \\ &= m · log\frac{1}{\sqrt{2\pi}\sigma} - \frac{1}{2} · \frac{1}{\sigma^2}\sum_{i=1}^{n}(y^{(i)} - \theta^Tx^{(i)})^2 \end{aligned}
logL(θ)=logi=1∏n2πσ1exp(−2σ2(y(i)−θTx(i))2)=i=1∑nlog2πσ1exp(−2σ2(y(i)−θTx(i))2)=m⋅log2πσ1−21⋅σ21i=1∑n(y(i)−θTx(i))2
上式求的是最大似然，值越大代表模型越好。减号前面是常数，不影响计算（舍去），只计算减号后的式子，所以此时的值越小代表模型越好。
J
(
θ
)
=
1
2
∑
i
=
1
n
(
y
(
i
)
−
θ
T
x
(
i
)
)
2
(
最
小
二
乘
法
)
=
1
2
∑
i
=
1
n
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
2
=
1
2
(
X
θ
−
y
)
T
(
X
θ
−
y
)
\begin{aligned} J(\theta) &= \frac{1}{2}\sum_{i=1}^{n} (y^{(i)} - \theta^Tx^{(i)})^2 \quad (最小二乘法) \\ &= \frac{1}{2}\sum_{i=1}^{n} (h_\theta(x^{(i)}) - y^{(i)})^2 \\ &= \frac{1}{2}(X\theta - y)^T(X\theta - y) \end{aligned}
J(θ)=21i=1∑n(y(i)−θTx(i))2(最小二乘法)=21i=1∑n(hθ(x(i))−y(i))2=21(Xθ−y)T(Xθ−y)
求偏导
∇
θ
J
(
θ
)
=
∇
θ
(
1
2
(
X
θ
−
y
)
T
(
X
θ
−
y
)
)
=
∇
θ
(
1
2
(
X
T
θ
T
−
y
T
)
(
X
θ
−
y
)
)
=
∇
θ
(
1
2
(
θ
T
X
T
X
θ
−
θ
T
X
T
y
−
y
T
X
θ
+
y
T
y
)
)
=
1
2
(
2
X
T
X
θ
−
X
T
y
−
(
y
T
X
)
T
)
=
X
T
X
θ
−
X
T
y
\begin{aligned} \nabla_\theta J(\theta) &= \nabla_\theta \left (\frac{1}{2}(X\theta - y)^T(X\theta - y) \right) \\ &= \nabla_\theta \left (\frac{1}{2}(X^T\theta^T - y^T)(X\theta - y) \right) \\ &= \nabla_\theta \left (\frac{1}{2}(\theta^TX^TX\theta - \theta^TX^Ty - y^TX\theta + y^Ty) \right) \\ &= \frac{1}{2} \left (2X^TX\theta - X^Ty - (y^TX)^T \right) \\ &= X^TX\theta - X^Ty \end{aligned}
∇θJ(θ)=∇θ(21(Xθ−y)T(Xθ−y))=∇θ(21(XTθT−yT)(Xθ−y))=∇θ(21(θTXTXθ−θTXTy−yTXθ+yTy))=21(2XTXθ−XTy−(yTX)T)=XTXθ−XTy
即：
∂
∂
θ
J
(
θ
)
=
∂
∂
θ
1
2
∑
i
=
1
n
(
h
θ
(
x
)
−
y
)
2
=
(
h
θ
(
x
)
−
y
)
x
(
i
)
\begin{aligned} \frac{\partial}{\partial\theta}J(\theta) &= \frac{\partial}{\partial\theta} \frac{1}{2} \sum_{i=1}^{n}(h_\theta(x) - y)^2 \\ &= (h_\theta(x) - y)x^{(i)} \end{aligned}
∂θ∂J(θ)=∂θ∂21i=1∑n(hθ(x)−y)2=(hθ(x)−y)x(i)
权重更新过程
θ
i
:
=
θ
i
−
α
∂
∂
θ
J
(
θ
)
=
θ
i
−
α
(
h
θ
(
x
)
−
y
)
x
(
i
)
\theta_i := \theta_i - \alpha \frac{\partial}{\partial\theta}J(\theta) = \theta_i - \alpha (h_\theta(x) - y)x^{(i)}
θi:=θi−α∂θ∂J(θ)=θi−α(hθ(x)−y)x(i)
根据如上 theta 的更新过程写一段 Python 代码，可以更直观的看到权重的拟合过程。
# 假设 y = theta * X，且已知 y = 30，X = 5，所以最终需学习出的 theta = 6。
# 但现在不知道theta 的值，所以设置初始 theta = 0，步长 alpha = 0.01，theta 的拟合过程如下
theta = 0
alpha = 0.01
X = 5
y = 30
# 你可以任意修改 alpha，theta，X，y 以及 for 循环的长度，来更直观的理解权重更新
for i in range(20):
theta = theta - alpha * (theta * X - y) * X
print(theta)
# theta 从 0 开始，一步步向 6 逼近
1.5
2.625
3.46875
4.1015625
4.576171875
4.93212890625
5.1990966796875
5.399322509765625
5.549491882324219
5.662118911743164
5.746589183807373
5.80994188785553
5.857456415891647
5.8930923119187355
5.919819233939052
5.939864425454289
5.9548983190907165
5.966173739318037
5.974630304488528
5.980972728366396
4万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
