θ0:=θ0−α1m∑i=1m(hθ(x(i))−y(i))x(i)0θj:=θj−α[1m∑i=1m(hθ(x(i))−y(i))x(i)j+λmθj]
θ
0
:=
θ
0
−
α
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
0
(
i
)
θ
j
:=
θ
j
−
α
[
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
j
(
i
)
+
λ
m
θ
j
]
}
对
θj
θ
j
的梯度下降公式进行整理变形:
θj:=(1−αλm)θj−α1m∑i=1m(hθ(x(i))−y(i))x(i)j
θ
j
:=
(
1
−
α
λ
m
)
θ
j
−
α
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
j
(
i
)
(注:
1−αλm<1
1
−
α
λ
m
<
1
)
正规方程: 线性回归的正规方程:
θ=(XTX)−1XTy
θ
=
(
X
T
X
)
−
1
X
T
y
正则化线性回归的正规方程:
θ=(XTX+λL)−1XTy
θ
=
(
X
T
X
+
λ
L
)
−
1
X
T
y
其中,
L
L
为一个(n+1)∗(n+1)矩阵,形式为
⎡⎣⎢⎢⎢⎢⎢⎢⎢011⋱1⎤⎦⎥⎥⎥⎥⎥⎥⎥
[
0
1
1
⋱
1
]
正则化逻辑回归
代价函数:
J(θ)=−[1m∑i=1my(i)log(hθ(x(i)))+(1−y(i))log(1−hθ(x(i)))]+λ2m∑j=1nθ2j
J
(
θ
)
=
−
[
1
m
∑
i
=
1
m
y
(
i
)
l
o
g
(
h
θ
(
x
(
i
)
)
)
+
(
1
−
y
(
i
)
)
l
o
g
(
1
−
h
θ
(
x
(
i
)
)
)
]
+
λ
2
m
∑
j
=
1
n
θ
j
2
梯度下降: Repeat {
θ0:=θ0−α1m∑i=1m(hθ(x(i))−y(i))x(i)0θj:=θj−α[1m∑i=1m(hθ(x(i))−y(i))x(i)j+λmθj]
θ
0
:=
θ
0
−
α
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
0
(
i
)
θ
j
:=
θ
j
−
α
[
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
j
(
i
)
+
λ
m
θ
j
]