Logistic Regression--Simplified Cost Function and Gradient Descent
1.Cost fuction
We can compress our cost function’s two conditional cases into one case:
C
o
s
t
(
h
θ
(
x
)
,
y
)
=
−
y
l
o
g
(
h
θ
(
x
)
)
−
(
1
−
y
)
l
o
g
(
1
−
h
θ
(
x
)
)
Cost(h_\theta(x),y)=-ylog(h_\theta(x))-(1-y)log(1-h_\theta(x))
Cost(hθ(x),y)=−ylog(hθ(x))−(1−y)log(1−hθ(x))Notice that when y is equal to 1, then the second term
(
1
−
y
)
log
(
1
−
h
θ
(
x
)
)
(1-y)\log(1-h_\theta(x))
(1−y)log(1−hθ(x))will be zero and will not affect the result. If y is equal to 0, then the first term
−
y
log
(
h
θ
(
x
)
)
-y \log(h_\theta(x))
−ylog(hθ(x))will be zero and will not affect the result.
We can fully write out our entire cost function as follows:
J
(
θ
)
=
−
1
m
∑
i
=
1
m
[
y
(
i
)
log
(
h
θ
(
x
(
i
)
)
)
+
(
1
−
y
(
i
)
)
log
(
1
−
h
θ
(
x
(
i
)
)
)
]
J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m [y^{(i)}\log (h_\theta (x^{(i)})) + (1 - y^{(i)})\log (1 - h_\theta(x^{(i)}))]
J(θ)=−m1i=1∑m[y(i)log(hθ(x(i)))+(1−y(i))log(1−hθ(x(i)))]A vectorized implementation is:
h
=
g
(
X
θ
)
h=g(Xθ)
h=g(Xθ)
J
(
θ
)
=
1
m
⋅
(
−
y
T
l
o
g
(
h
)
−
(
1
−
y
)
T
l
o
g
(
1
−
h
)
)
J(θ)=1m⋅(−y^Tlog(h)−(1−y)^Tlog(1−h))
J(θ)=1m⋅(−yTlog(h)−(1−y)Tlog(1−h))
2.Gradient Descent
Remember that the general form of gradient descent is:
R
e
p
e
a
t
{
θ
j
:
=
θ
j
−
α
∂
∂
θ
j
J
(
θ
)
}
Repeat \{ \theta_j:=\theta_j-\alpha \frac{\partial }{\partial \theta_j}J(\theta) \}
Repeat{θj:=θj−α∂θj∂J(θ)}We can work out the derivative part using calculus to get:
R
e
p
e
a
t
{
θ
j
:
=
θ
j
−
α
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
−
y
(
i
)
)
x
j
(
i
)
}
Repeat \{ \theta_j:=\theta_j-\frac{\alpha}{m} \sum_{i=1}^{m}(h_{\theta}(x^{(i)}-y^{(i)}) x_j^{(i)} \}
Repeat{θj:=θj−mαi=1∑m(hθ(x(i)−y(i))xj(i)}Notice that this algorithm is identical to the one we used in linear regression. We still have to simultaneously update all values in theta.
A vectorized implementation is:
θ
:
=
t
h
e
t
a
−
α
m
(
g
(
X
θ
)
−
y
⃗
)
]
\theta:=theta-\frac{\alpha}{m}(g(X\theta)-\vec{y})]
θ:=theta−mα(g(Xθ)−y)]
3.supplementary
求导的过程还在推导
1170
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
