机器学习算法完整版见fenghaootong-github
贝叶斯(Bayesian Techniques)
- 朴素贝叶斯
- 高斯贝叶斯分类器
- 多项式贝叶斯分类器
- 伯努利贝叶斯分类器
贝叶斯定理:
P(Bi|A)=P(A|Bi)P(B)∑nj=1P(A|Bi)P(Bj) P ( B i | A ) = P ( A | B i ) P ( B ) ∑ j = 1 n P ( A | B i ) P ( B j )
朴素贝叶斯方法
原理:
- 先验概率分布: P(Y=ck),k=1,2,...,K. P ( Y = c k ) , k = 1 , 2 , . . . , K .
- 条件概率分布: P(X=x⃗ |Y=ck)=P(X1=x1,X2=x2,...,Xn=xn|Y=ck)=Πnj=1P(Xi=xi|Y=ck),k=1,2,...,K P ( X = x → | Y = c k ) = P ( X 1 = x 1 , X 2 = x 2 , . . . , X n = x n | Y = c k ) = Π j = 1 n P ( X i = x i | Y = c k ) , k = 1 , 2 , . . . , K
根据贝叶斯定理:
P(Y=ck|X=x⃗ )=P(X=x⃗ |Y=ck)P(Y=ck)∑Kj=1P(X=x⃗ |Y=cj)P(Y=cj) P ( Y = c k | X = x → ) = P ( X = x → | Y = c k ) P ( Y = c k ) ∑ j = 1 K P ( X = x → | Y = c j ) P ( Y = c j )
考虑分类特征的条件独立性假设有:
P(Y=ck|X=x⃗ )=P(Y=ck)Πnj=1P(Xi=xi|Y=ck)∑Kj=1P(X=x⃗ |Y=cj)P(Y=cj),k=1,2,...,K P ( Y = c k | X = x → ) = P ( Y = c k ) Π j = 1 n P ( X i = x i | Y = c k ) ∑ j = 1 K P ( X = x → | Y = c j ) P ( Y = c j ) , k = 1 , 2 , . . . , K
于是朴素贝叶斯分类器表示为:
y=f(x⃗ )=argmaxckP(Y=ck)Πnj=1P(Xi=xi|Y=ck)∑Kj=1P(X=x⃗ |Y=cj)P(Y=cj) y = f ( x → ) = a r g max c k P ( Y = c k ) Π j = 1 n P ( X i = x i | Y = c k ) ∑ j = 1 K P ( X = x → | Y = c j ) P ( Y = c j )
对于所有的 ck c k ,上面的分母一样,所以:
y=f(x⃗ )=argmaxckP(Y=ck)Πnj=1P(Xi=xi|Y=ck) y = f ( x → ) = a r g max c k P ( Y = c k ) Π j = 1 n P ( X i = x i | Y = c k )
朴素贝叶斯算法
输入:
- 训练集T= (x1→,y1),(x2→,y2),...,(xN→,yN),xi→=(x(1)i,x(2)i,...,x(n)i)T,x(j)i ( x 1 → , y 1 ) , ( x 2 → , y 2 ) , . . . , ( x N → , y N ) , x i → = ( x i ( 1 ) , x i ( 2 ) , . . . , x i ( n ) ) T , x i ( j ) 为第i个样本的第j个特征,其中 x(j)i∈aj1,aj2,...,ajsj,ajl x i ( j ) ∈ a j 1 , a j 2 , . . . , a j s j , a j l 为第j个特征可能取到的第l个值
算法步骤
- 计算先验概率的估计值以及条件概率的估计值
P(Y=ck)=∑Ni=1I(yi=ck)N,k=1,2,...,K
P
(
Y
=
c
k
)
=
∑
i
=
1
N
I
(
y
i
=
c
k
)
N
,
k
=
1
,
2
,
.
.
.
,
K
P(X(j)=ajl|Y=ck)=∑Ni=1I(x(j)i=ajl,yi=ck)∑Ni=1I(yi=ck)
P
(
X
(
j
)
=
a
j
l
|
Y
=
c
k
)
=
∑
i
=
1
N
I
(
x
i
(
j
)
=
a
j
l
,
y
i
=
c
k
)
∑
i
=
1
N
I
(
y
i
=
c
k
)
j=1,2,...,n;l=1,2,...,sj;k=1,2,...,K
j
=
1
,
2
,
.
.
.
,
n
;
l
=
1
,
2
,
.
.
.
,
s
j
;
k
=
1
,
2
,
.
.
.
,
K
- 对于给定的实例 x⃗ =(x(1),x(2),...,x(n))T x → = ( x ( 1 ) , x ( 2 ) , . . . , x ( n ) ) T ,计算
P(Y=ck)Πnj=1P(Xi=xi|Y=ck),k=1,2,...,K P ( Y = c k ) Π j = 1 n P ( X i = x i | Y = c k ) , k = 1 , 2 , . . . , K
- 计算给并返回实例 x⃗ x → 的分类y:
y=f(x⃗ )=argmaxckP(Y=ck)Πnj=1P(Xi=xi|Y=ck) y = f ( x → ) = a r g max c k P ( Y = c k ) Π j = 1 n P ( X i = x i | Y = c k )
贝叶斯估计
条件概率 P(X(j)=ajl|Y=ck) P ( X ( j ) = a j l | Y = c k ) 的极大似然估计:
P(X(j)=ajl|Y=ck)=∑Ni=1I(x(j)i=ajl,yi=ck)∑Ni=1I(yi=ck)
P
(
X
(
j
)
=
a
j
l
|
Y
=
c
k
)
=
∑
i
=
1
N
I
(
x
i
(
j
)
=
a
j
l
,
y
i
=
c
k
)
∑
i
=
1
N
I
(
y
i
=
c
k
)
j=1,2,...,n;
j
=
1
,
2
,
.
.
.
,
n
;
l=1,2,...,sj;
l
=
1
,
2
,
.
.
.
,
s
j
;
k=1,2,...,K
k
=
1
,
2
,
.
.
.
,
K
用极大似然估计可能会出现分母为零的情况,此时可以采用贝叶斯估计:
P(X(j)=ajl|Y=ck)=∑Ni=1I(x(j)i=ajl,yi=ck)+λ∑Ni=1I(yi=ck)+sjλ
P
(
X
(
j
)
=
a
j
l
|
Y
=
c
k
)
=
∑
i
=
1
N
I
(
x
i
(
j
)
=
a
j
l
,
y
i
=
c
k
)
+
λ
∑
i
=
1
N
I
(
y
i
=
c
k
)
+
s
j
λ
j=1,2,...,n;
j
=
1
,
2
,
.
.
.
,
n
;
l=1,2,...,sj;
l
=
1
,
2
,
.
.
.
,
s
j
;
k=1,2,...,K
k
=
1
,
2
,
.
.
.
,
K
它满足概率分布函数的条件:
Pλ(X(j)=ajl|Y=ck)>0,l=1,2,...,sj;k=1,2,...,K
P
λ
(
X
(
j
)
=
a
j
l
|
Y
=
c
k
)
>
0
,
l
=
1
,
2
,
.
.
.
,
s
j
;
k
=
1
,
2
,
.
.
.
,
K
∑sjl=1Pλ(X(j)=ajl|Y=ck)=1
∑
l
=
1
s
j
P
λ
(
X
(
j
)
=
a
j
l
|
Y
=
c
k
)
=
1
此时 P(Y=ck) P ( Y = c k ) 的贝叶斯估计调整为:
Pλ(Y=ck)=∑Ni=1I(yi=ck)+λN+Kλ P λ ( Y = c k ) = ∑ i = 1 N I ( y i = c k ) + λ N + K λ
当 λ=0 λ = 0 时,为极大似然估计
当 λ=1 λ = 1 时,为拉普拉斯平滑
高斯贝叶斯分类器
它假设特征的条件概率分布满足高斯分布:
P(X(j)|y=ck)=12πσ2k√exp(−(X(j)−μk)22σ2k) P ( X ( j ) | y = c k ) = 1 2 π σ k 2 e x p ( − ( X ( j ) − μ k ) 2 2 σ k 2 )
多项式贝叶斯分类器
它假设特征的条件概率分布满足多项式分布:
P(X(j)|y=ck)=Nkj+αNk+αn P ( X ( j ) | y = c k ) = N k j + α N k + α n
伯努利贝叶斯分类器
它假设特征的条件概率分布满足伯努利分布:
P(X(j)|y=ck)=pX(j)+(1−p)(1−X(j)) P ( X ( j ) | y = c k ) = p X ( j ) + ( 1 − p ) ( 1 − X ( j ) )
Bayes应用实例