Week Six
F Score
P = 2 1 P + 1 R = 2 P R P + R \begin{aligned} P &= &\dfrac{2}{\dfrac{1}{P}+\dfrac{1}{R}}\\ &= &2 \dfrac{PR}{P+R} \end{aligned} P==P1+R122P+RPR
Week Seven
Support Vector Machine
Cost Function
min
θ
[
−
1
m
∑
y
i
∈
Y
,
x
i
∈
X
y
i
log
h
(
θ
T
x
i
)
+
(
1
−
y
i
)
log
(
1
−
h
(
θ
T
x
i
)
)
+
λ
2
m
∑
θ
i
∈
θ
θ
i
2
]
⇒
min
θ
[
−
∑
y
i
∈
Y
,
x
i
∈
X
y
i
log
h
(
θ
T
x
i
)
+
(
1
−
y
i
)
log
(
1
−
h
(
θ
T
x
i
)
)
+
λ
2
∑
θ
i
∈
θ
θ
i
2
]
⇒
min
θ
[
C
∑
y
i
∈
Y
,
x
i
∈
X
y
i
log
h
(
θ
T
x
i
)
+
(
1
−
y
i
)
log
(
1
−
h
(
θ
T
x
i
)
)
+
∑
θ
i
∈
θ
θ
i
2
]
\begin{aligned} &\min_{\theta}\lbrack-\dfrac{1}{m}{\sum_{y_{i}\in Y, x_{i} \in X}{y_{i} \log h(\theta^{T}x_{i})}+(1-y_{i})\log (1-h(\theta^{T}x_{i}))+\dfrac{\lambda}{2m} \sum_{\theta_{i} \in \theta}{\theta_{i}^{2}}}\rbrack\\ &\Rightarrow \min_{\theta}[-\sum_{y_{i} \in Y,x_{i} \in X}{y_{i} \log{h(\theta^{T}x_{i})}+(1-y_{i})\log(1-h(\theta^{T}x_{i}}))+\dfrac{\lambda}{2}\sum_{\theta_{i} \in \theta }{\theta^2_{i}}]\\ &\Rightarrow\min_{\theta}[C\sum_{y_{i} \in Y,x_{i} \in X}{y_{i} \log{h(\theta^{T}x_{i})}+(1-y_{i})\log(1-h(\theta^{T}x_{i}}))+\sum_{\theta_{i} \in \theta }{\theta^2_{i}}]\\ \end{aligned}
θmin[−m1yi∈Y,xi∈X∑yilogh(θTxi)+(1−yi)log(1−h(θTxi))+2mλθi∈θ∑θi2]⇒θmin[−yi∈Y,xi∈X∑yilogh(θTxi)+(1−yi)log(1−h(θTxi))+2λθi∈θ∑θi2]⇒θmin[Cyi∈Y,xi∈X∑yilogh(θTxi)+(1−yi)log(1−h(θTxi))+θi∈θ∑θi2]
C is somewhat
1
λ
\dfrac{1}{\lambda}
λ1.
- Large C:
- lower bias, high variance
- Small C:
- Higher bias, low variance
- Large
σ
2
\sigma^2
σ2: Features
f
i
f_{i}
fi vary more smoothly.
- Higher bias, low variance
- Small
σ
2
\sigma^2
σ2: Features
f
i
f_{i}
fi vary more sharply.
- Lower bias, high variance.
1 2 ∑ θ i ∈ θ θ i 2 s . t θ T x i ≥ 1 , i f y i = 1 θ T x i ≤ − 1 , i f y i = 0 \begin{aligned} & \dfrac{1}{2} \sum_{\theta_{i} \in \theta}{\theta_{i}^2}\\ &s.t&\theta^{T}x_{i} \geq 1, if\ y_{i} = 1&\\ &&\theta^{T}x_{i} \leq -1, if\ y_{i} = 0& \end{aligned} 21θi∈θ∑θi2s.tθTxi≥1,if yi=1θTxi≤−1,if yi=0
- Lower bias, high variance.
PS
If features are too many related to m, use logistic regression or SVM without a kernel.
If n is small, m is intermediate, use SVM with Gaussian kernal.
If n is small, m is large, add more features and use logistic regression or SVM without a kernel.
Week Eight
K-means
Cost Function
It try to minimize
min
μ
1
m
∑
i
=
1
m
∣
∣
x
(
i
)
−
μ
c
(
i
)
∣
∣
2
\min_{\mu}{\dfrac{1}{m} \sum_{i=1}^{m} ||x^{(i)} - \mu_{c^{(i)}}}||^2
μminm1i=1∑m∣∣x(i)−μc(i)∣∣2
For the first loop, minimize the cost function by varing the centorid. For the second loop, it minimize the cost funcion with cetorid fixed and realign the centorid of every x in the training set.
Initialize
Initialize the centorids randomly. Randomly select k samples from the training set and set the centorids to these random selected samples.
It is possible that K-meas fall into the local minimum, So repeat to initialize the centorids randomly until the cost(distortion) is suitable for your purposes.
K-means converge all the time and it will not increase the cost during the training processs. More centoirds will decease the cost, if not, the k-means must fall into the local minimum and reinitialize the centorid until the cost is less.
PCA (Principal Component Analysis)
Restruct x from z meeting the below nonequation
1
−
1
m
∑
i
=
1
m
∣
∣
x
(
i
)
−
x
a
p
p
r
o
x
i
m
a
t
i
o
n
(
i
)
∣
∣
2
1
m
∑
i
=
1
m
∣
∣
x
(
i
)
∣
∣
2
≥
0.99
1-\dfrac{\dfrac{1}{m} \sum_{i=1}^{m}||x^{(i)}-x^{(i)}_{approximation}||^2}{\dfrac{1}{m} \sum_{i=1}^{m} ||x^{(i)}||^2} \geq 0.99
1−m1∑i=1m∣∣x(i)∣∣2m1∑i=1m∣∣x(i)−xapproximation(i)∣∣2≥0.99
PS:
the nonequation can be equal to the below
[
U
,
S
,
D
]
=
s
v
d
(
s
i
g
m
a
)
U
r
e
d
u
c
e
=
U
(
:
,
1
:
k
)
z
=
U
r
e
d
u
c
e
′
∗
x
x
a
p
p
r
o
x
i
m
a
t
i
o
n
=
U
r
e
d
u
c
e
∗
x
S
=
(
s
11
0
⋯
0
0
s
22
⋯
0
⋮
⋮
⋱
⋮
0
0
⋯
s
n
n
)
∑
i
=
1
k
s
i
i
2
∑
i
=
1
m
s
i
i
2
≥
0.99
\begin{aligned} [U, S, D] &= svd(sigma)\\ U_{reduce} &= U(:, 1:k)\\ z &= U_{reduce}' * x\\ x_{approximation} &= U_{reduce} * x\\\\ S &= \left( \begin{array}{ccc} s_{11}&0&\cdots&0\\ 0&s_{22}&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&s_{nn} \end{array} \right)\\\\ \dfrac{\sum_{i=1}^{k}s_{ii}^2}{\sum_{i=1}^{m} s_{ii}^2} &\geq 0.99 \end{aligned}
[U,S,D]UreducezxapproximationS∑i=1msii2∑i=1ksii2=svd(sigma)=U(:,1:k)=Ureduce′∗x=Ureduce∗x=⎝⎜⎜⎜⎛s110⋮00s22⋮0⋯⋯⋱⋯00⋮snn⎠⎟⎟⎟⎞≥0.99
Week Nine
Anomaly Detection
Gaussian Distribution
Multivariate Gaussian Distribution takes the connection of different variants into account
p
(
x
)
=
1
(
2
π
)
n
2
∣
Σ
∣
1
2
e
−
1
2
(
x
−
μ
)
T
Σ
−
1
(
x
−
μ
)
p(x) = \dfrac{1}{(2\pi)^{\frac{n}{2}}|\Sigma|^{\frac{1}{2}}}e^{-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)}
p(x)=(2π)2n∣Σ∣211e−21(x−μ)TΣ−1(x−μ)
Single variant Gaussian Distribution is a special example of Multivariate Gaussian Distribution, where
Σ
=
(
σ
11
σ
22
⋱
σ
n
n
)
\Sigma = \left(\begin{array}{ccc} \sigma_{11}&&&&\\ &\sigma_{22}&&&\\ &&\ddots&&\\ &&&\sigma_{nn}&\\ \end{array}\right)
Σ=⎝⎜⎜⎛σ11σ22⋱σnn⎠⎟⎟⎞
When training the Anomaly Detection, we can use Maximum Likelihood Estimation
μ
=
1
m
∑
i
=
1
m
x
(
i
)
Σ
=
1
m
∑
i
=
1
m
(
x
(
i
)
−
μ
)
(
x
(
i
)
−
μ
)
T
\begin{aligned} \mu &= \dfrac{1}{m} \sum_{i=1}^{m}x^{(i)}\\ \Sigma &= \dfrac{1}{m} \sum_{i=1}^{m} (x^{(i)}-\mu)(x^{(i)}-\mu)^{T} \end{aligned}
μΣ=m1i=1∑mx(i)=m1i=1∑m(x(i)−μ)(x(i)−μ)T
When we use single variant anomaly detection, the numerical cost is much cheaper than multivariant. But may need to add some new features to distinguish the normal and non-normal.
Recommender System
Cost Function
J
(
X
,
Θ
)
=
1
2
∑
(
i
,
j
)
:
r
(
i
,
j
)
=
1
(
(
θ
(
j
)
)
T
x
(
i
)
−
y
(
i
,
j
)
)
2
+
λ
2
[
∑
i
=
1
n
m
∑
k
=
1
n
(
x
k
(
i
)
)
2
+
∑
j
=
1
n
μ
∑
k
=
1
n
(
θ
k
(
j
)
)
2
]
J
(
X
,
Θ
)
=
1
2
S
u
m
{
(
X
Θ
′
−
Y
)
.
∗
R
}
+
λ
2
(
S
u
m
{
Θ
.
2
}
+
S
u
m
{
X
.
2
}
\begin{aligned} J(X,\Theta) = \dfrac{1}{2} \sum_{(i,j):r(i,j)=1}((\theta^{(j)})^{T}x^{(i)}-y^{(i,j)})^2 + \dfrac{\lambda}{2}[\sum_{i=1}^{n_{m}}\sum_{k=1}^{n}(x_k^{(i)})^2 + \sum_{j=1}^{n_\mu} \sum_{k=1}^n(\theta_{k}^{(j)})^2]\\ J(X,\Theta) = \dfrac{1}{2}Sum\{(X\Theta'-Y).*R\} + \dfrac{\lambda}{2}(Sum\{\Theta.^2\} + Sum\{X.^2\}\\ \end{aligned}
J(X,Θ)=21(i,j):r(i,j)=1∑((θ(j))Tx(i)−y(i,j))2+2λ[i=1∑nmk=1∑n(xk(i))2+j=1∑nμk=1∑n(θk(j))2]J(X,Θ)=21Sum{(XΘ′−Y).∗R}+2λ(Sum{Θ.2}+Sum{X.2}
∂
J
∂
X
=
(
(
X
Θ
′
−
Y
)
.
∗
R
)
Θ
+
λ
X
∂
J
∂
Θ
=
(
(
X
Θ
′
−
Y
)
.
∗
R
)
′
X
+
λ
Θ
\begin{aligned} \dfrac{\partial J}{\partial X} = ((X\Theta'-Y).*R) \Theta + \lambda X\\ \dfrac{\partial J}{\partial \Theta} = ((X\Theta'-Y).*R)'X + \lambda \Theta \end{aligned}
∂X∂J=((XΘ′−Y).∗R)Θ+λX∂Θ∂J=((XΘ′−Y).∗R)′X+λΘ