机器学习——朴素贝叶斯法
朴素贝叶斯(naive Bayes)法是 基于贝叶斯定理与特征条件独立假设的分类方法。首先基于特征条件独立假设 学习输入输出的联合概率分布,然后基于此模型,对给定的输入 x x x,利用贝叶斯定理 求出后验概率最大的输出 y y y。
基本方法
通过训练数据集学习联合概率分布
P
(
X
,
Y
)
P(X,Y)
P(X,Y),Based on先验概率
P
(
Y
=
c
k
)
P(Y=c_k)
P(Y=ck)和条件概率
P
(
X
=
x
∣
Y
=
c
k
)
P \left( X = x \mid Y = c _ { k } \right)
P(X=x∣Y=ck)求得。
先验概率分布
P
(
Y
=
c
k
)
,
k
=
1
,
2
,
⋯
,
K
P(Y=c_k), \quad k=1,2, \cdots, K
P(Y=ck),k=1,2,⋯,K
条件概率分布
P
(
X
=
x
∣
Y
=
c
k
)
=
P
(
X
(
1
)
=
x
(
1
)
,
⋯
,
X
(
n
)
=
x
(
n
)
∣
Y
=
c
k
)
,
k
=
1
,
2
,
⋯
,
K
P \left( X = x \mid Y = c _ { k } \right) = P \left( X ^ { ( 1 ) } = x ^ { ( 1 ) } , \cdots , X ^ { ( n ) } = x ^ { ( n ) } \mid Y = c _ { k } \right) , \quad k = 1,2 , \cdots , K
P(X=x∣Y=ck)=P(X(1)=x(1),⋯,X(n)=x(n)∣Y=ck),k=1,2,⋯,K
于是学习到联合概率分布
P
(
X
,
Y
)
P(X,Y)
P(X,Y)。
朴素贝叶斯法对条件概率分布做了条件独立性的假设,该假设是:
P
(
X
=
x
∣
Y
=
c
k
)
=
P
(
X
(
1
)
=
x
(
1
)
,
⋯
,
X
(
n
)
=
x
(
n
)
∣
Y
=
c
k
)
=
∏
j
=
1
n
P
(
X
(
j
)
=
x
(
j
)
∣
Y
=
c
k
)
\begin{aligned} P \left( X = x \mid Y = c _ { k } \right) & = P \left( X ^ { ( 1 ) } = x ^ { ( 1 ) } , \cdots , X ^ { ( n ) } = x ^ { ( n ) } \mid Y = c _ { k } \right) \\ & = \prod _ { j = 1 } ^ { n } P \left( X ^ { ( j ) } = x ^ { ( j ) } \mid Y = c _ { k } \right) \end{aligned}
P(X=x∣Y=ck)=P(X(1)=x(1),⋯,X(n)=x(n)∣Y=ck)=j=1∏nP(X(j)=x(j)∣Y=ck)
条件独立假设等于是说用于分类的特征在类确定的条件下都是条件独立的。
后验概率计算根据贝叶斯定理进行:
P
(
Y
=
c
k
∣
X
=
x
)
=
P
(
X
=
x
∣
Y
=
c
k
)
P
(
Y
=
c
k
)
∑
k
P
(
X
=
x
∣
Y
=
c
k
)
P
(
Y
=
c
k
)
P \left( Y = c _ { k } \mid X = x \right) = \frac { P \left( X = x \mid Y = c _ { k } \right) P \left( Y = c _ { k } \right) } { \sum _ { k } P \left( X = x \mid Y = c _ { k } \right) P \left( Y = c _ { k } \right) }
P(Y=ck∣X=x)=∑kP(X=x∣Y=ck)P(Y=ck)P(X=x∣Y=ck)P(Y=ck)
于是朴素贝叶斯分类器可表示为
y
=
f
(
x
)
=
arg
max
c
k
P
(
Y
=
c
k
)
∏
j
P
(
X
(
j
)
=
x
(
j
)
∣
Y
=
c
k
)
∑
k
P
(
Y
=
c
k
)
∏
j
P
(
X
(
j
)
=
x
(
j
)
∣
Y
=
c
k
)
y = f ( x ) = \arg \max _ { c _ { k } } \frac { P \left( Y = c _ { k } \right) \prod _ { j } P \left( X ^ { ( j ) } = x ^ { ( j ) } \mid Y = c _ { k } \right) } { \sum _ { k } P \left( Y = c _ { k } \right) \prod _ { j } P \left( X ^ { ( j ) } = x ^ { ( j ) } \mid Y = c _ { k } \right) }
y=f(x)=argckmax∑kP(Y=ck)∏jP(X(j)=x(j)∣Y=ck)P(Y=ck)∏jP(X(j)=x(j)∣Y=ck)
由于分母对所有
c
k
c_k
ck都是相同的,所以可以只保留分子。
朴素贝叶斯的参数估计
学习意味着估计
P
(
Y
=
c
k
)
P(Y=c_k)
P(Y=ck)和
P
(
X
(
j
)
=
x
(
j
)
∣
Y
=
c
k
)
P \left( X ^ { ( j ) } = x ^ { ( j ) } \mid Y = c _ { k } \right)
P(X(j)=x(j)∣Y=ck),可以用极大似然估计法估计相应的概率。先验概率
P
(
Y
=
c
k
)
P(Y=c_k)
P(Y=ck)的极大似然估计是
P
(
Y
=
c
k
)
=
∑
i
=
1
N
I
(
y
i
=
c
k
)
N
,
k
=
1
,
2
,
⋯
,
K
P \left( Y = c _ { k } \right) = \frac { \sum _ { i = 1 } ^ { N } I \left( y _ { i } = c _ { k } \right) } { N } , \quad k = 1,2 , \cdots , K
P(Y=ck)=N∑i=1NI(yi=ck),k=1,2,⋯,K
条件概率
P
(
X
(
j
)
=
a
j
l
∣
Y
=
c
k
)
P \left( X ^ { ( j ) } = a _ { j l } \mid Y = \right.c_k)
P(X(j)=ajl∣Y=ck)的极大似然估计是
P
(
X
(
j
)
=
a
j
l
∣
Y
=
c
k
)
=
∑
i
=
1
N
I
(
x
i
(
j
)
=
a
j
l
,
y
i
=
c
k
)
∑
i
=
1
N
I
(
y
i
=
c
k
)
j
=
1
,
2
,
⋯
,
n
;
l
=
1
,
2
,
⋯
,
S
j
;
k
=
1
,
2
,
⋯
,
K
\begin{array} { l } P \left( X ^ { ( j ) } = a _ { j l } \mid Y = c _ { k } \right) = \frac { \sum _ { i = 1 } ^ { N } I \left( x _ { i } ^ { ( j ) } = a _ { j l } , y _ { i } = c _ { k } \right) } { \sum _ { i = 1 } ^ { N } I \left( y _ { i } = c _ { k } \right) } \\ j = 1,2 , \cdots , n ; \quad l = 1,2 , \cdots , S _ { j } ; \quad k = 1,2 , \cdots , K \end{array}
P(X(j)=ajl∣Y=ck)=∑i=1NI(yi=ck)∑i=1NI(xi(j)=ajl,yi=ck)j=1,2,⋯,n;l=1,2,⋯,Sj;k=1,2,⋯,K
贝叶斯估计
极大似然估计可能会出现所要估计的概率值为0的情况。解决这一问题的方法是采用贝叶斯估计。
P
λ
(
X
(
j
)
=
a
j
l
∣
Y
=
c
k
)
=
∑
i
=
1
N
I
(
x
i
(
j
)
=
a
j
l
,
y
i
=
c
k
)
+
λ
∑
i
=
1
N
I
(
y
i
=
c
k
)
+
S
j
λ
P _ { \lambda } \left( X ^ { ( j ) } = a _ { j l } \mid Y = c _ { k } \right) = \frac { \sum _ { i = 1 } ^ { N } I \left( x _ { i } ^ { ( j ) } = a _ { j l } , y _ { i } = c _ { k } \right) + \lambda } { \sum _ { i = 1 } ^ { N } I \left( y _ { i } = c _ { k } \right) + S _ { j } \lambda }
Pλ(X(j)=ajl∣Y=ck)=∑i=1NI(yi=ck)+Sjλ∑i=1NI(xi(j)=ajl,yi=ck)+λ
常取
λ
=
1
\lambda = 1
λ=1,这时称为拉普拉斯平滑(Laplacian smoothing)。显然有
P
λ
(
X
(
j
)
=
a
j
l
∣
Y
=
c
k
)
>
0
∑
l
=
1
S
j
P
(
X
(
j
)
=
a
j
l
∣
Y
=
c
k
)
=
1
\begin{array} { l } P _ { \lambda } \left( X ^ { ( j ) } = a _ { j l } \mid Y = c _ { k } \right) > 0 \\ \sum _ { l = 1 } ^ { S _ { j } } P \left( X ^ { ( j ) } = a _ { j l } \mid Y = c _ { k } \right) = 1 \end{array}
Pλ(X(j)=ajl∣Y=ck)>0∑l=1SjP(X(j)=ajl∣Y=ck)=1
同样先验概率的贝叶斯估计是
P
λ
(
Y
=
c
k
)
=
∑
i
=
1
N
I
(
y
i
=
c
k
)
+
λ
N
+
K
λ
P _ { \lambda } \left( Y = c _ { k } \right) = \frac { \sum _ { i = 1 } ^ { N } I \left( y _ { i } = c _ { k } \right) + \lambda } { N + K \lambda }
Pλ(Y=ck)=N+Kλ∑i=1NI(yi=ck)+λ