朴素贝叶斯的学习与分类
-
基本方法
定义:设输入空间 χ ⊆ R n \chi\subseteq R^n χ⊆Rn为n维向量的集合,输出空间为类标记集合 γ = { c 1 , c 2 , . . , c k } . \gamma=\{c_1,c_2,..,c_k\}. γ={c1,c2,..,ck}.输入的特征向量 x ∈ χ , x \in \chi , x∈χ,输出类别 y ∈ γ . X y\in\gamma .X y∈γ.X是定义在输入空间 χ \chi χ上的随机向量, Y Y Y是定义在输出空间 γ \gamma γ上的随机变量。 P ( X , Y ) P(X,Y) P(X,Y)是 X X X和 Y Y Y的联合概率分布.训练数据集
T = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x N , y N ) } T=\{(x_1,y_1),(x_2,y_2),...,(x_N,y_N)\} T={(x1,y1),(x2,y2),...,(xN,yN)}由 P ( X , Y ) P(X,Y) P(X,Y)独立同分布产生.
朴素贝叶斯通过训练数据生成 P ( X , Y ) . P(X,Y). P(X,Y). 具体地,学习以下两个先验概率分布
P ( Y = c k ) , k = 1 , 2 , . . . , K P(Y=c_k),k=1,2,...,K P(Y=ck),k=1,2,...,K
P ( X = x ∣ Y = c k ) = P ( X ( 1 ) = x ( 1 ) , . . . , X ( n ) = x ( n ) ∣ Y = c k ) , k = 1 , 2 , . . , K P(X=x|Y=c_k)=P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}|Y=c_k),k=1,2,..,K P(X=x∣Y=ck)=P(X(1)=x(1),...,X(n)=x(n)∣Y=ck),k=1,2,..,K
于是我们就学到联合概率分布,因为参数很多,我们假设所有条件独立。
P ( X = x ∣ Y = c k ) = P ( X ( 1 ) = x ( 1 ) , . . . , X ( n ) = x ( n ) ∣ Y = c k ) = ∏ i = 1 N P ( X ( i ) = x ( i ) ∣ Y = c k ) P(X=x|Y=c_k)=P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}|Y=c_k)=\prod_{i=1}^N P(X^{(i)}=x^{(i)}|Y=c_k) P(X=x∣Y=ck)=P(X(1)=x(1),...,X(n)=x(n)∣Y=ck)=∏i=1NP(X(i)=x(i)∣Y=ck)
因此对于实例我们可以这样预测
P ( Y = c k ∣ X = x ) = P ( X = x ∣ Y = c k ) ∗ P ( Y = c k ) ∑ i P ( X = x ∣ Y = c i ) ∗ P ( Y = c i ) P(Y=c_k|X=x)=\frac{P(X=x|Y=c_k)*P(Y=c_k)}{\sum\limits_iP(X=x|Y=c_i)*P(Y=c_i)} P(Y=ck∣X=x)=i∑P(X=x∣Y=ci)∗P(Y=ci)P(X=x∣Y=ck)∗P(Y=ck)
P ( Y = c k ∣ X = x ) = ∏ i = 1 N P ( X ( i ) = x ( i ) ∣ Y = c k ) ∗ P ( Y = c k ) ∑ i ∏ i = 1 N P ( X ( j ) = x ( j ) ∣ Y = c k ) ∗ P ( Y = c k ) , k = 1 , 2 , . . . , K P(Y=c_k|X=x)=\frac{\prod_{i=1}^N P(X^{(i)}=x^{(i)}|Y=c_k)*P(Y=c_k)}{\sum\limits_i\prod_{i=1}^N P(X^{(j)}=x^{(j)}|Y=c_k)*P(Y=c_k)},k=1,2,...,K P(Y=ck∣X=x)=i∑∏i=1NP(X(j)=x(j)∣Y=ck)∗P(Y=ck)∏i=1NP(X(i)=x(i)∣Y=ck)∗P(Y=ck),k=1,2,...,K
y = f ( x ) = a r g m a x c k = ∏ i = 1 N P ( X ( i ) = x ( i ) ∣ Y = c k ) ∗ P ( Y = c k ) ∑ i ∏ i = 1 N P ( X ( j ) = x ( j ) ∣ Y = c k ) ∗ P ( Y = c k ) y=f(x)=argmax_{c_k}=\frac{\prod_{i=1}^N P(X^{(i)}=x^{(i)}|Y=c_k)*P(Y=c_k)}{\sum\limits_i\prod_{i=1}^N P(X^{(j)}=x^{(j)}|Y=c_k)*P(Y=c_k)} y=f(x)=argmaxck=i∑∏i=1NP(X(j)=x(j)∣Y=ck)∗P(Y=ck)∏i=1NP(X(i)=x(i)∣Y=ck)∗P(Y=ck)
y = f ( x ) = a r g m a x c k P ( Y = c k ) ∗ ∏ i = 1 N P ( X ( i ) = x ( i ) ∣ Y = c k ) y=f(x)=argmax_{c_k}P(Y=c_k)*\prod_{i=1}^N P(X^{(i)}=x^{(i)}|Y=c_k) y=f(x)=argmaxckP(Y=ck)∗∏i=1NP(X(i)=x(i)∣Y=ck) -
后验概率最大化的含义
后验概率最大类,等价于期望风险最小化。假设选择 0 − 1 0-1 0−1损失函数
L ( Y , f ( X ) ) = { + 1 Y ≠ f ( X ) − 1 Y = f ( X ) L(Y,f(X))=\begin{cases} +1 &Y\ne f(X)\\ -1 &Y= f(X)\\ \end{cases} L(Y,f(X))={+1−1Y=f(X)Y=f(X)
这里 f ( X ) f(X) f(X)是决策函数,这时期望风险函数为 R e x p ( f ) = E [ L ( Y , f ( X ) ) ] R_{exp}(f)=E[L(Y,f(X))] Rexp(f)=E[L(Y,f(X))]其为对联合概率分布 P ( X , Y ) P(X,Y) P(X,Y)取的
所以 R e x p ( f ) = E X [ ∑ k = 1 K [ L ( c k , f ( X ) ) ] ∗ P ( c k ∣ X ) ] R_{exp}(f)=E_{X}[\sum\limits_{k=1}^K[L(c_k,f(X))]*P(c_k|X)] Rexp(f)=EX[k=1∑K[L(ck,f(X))]∗P(ck∣X)],为了其最小化只需针对每个 x x x最小化就好了
f ( x ) = a r g m i n y ∈ γ ∑ k = 1 K L ( c k , y ) ∗ P ( c k ∣ X = x ) = a r g m i n y ∈ γ ∑ k = 1 K P ( y ≠ c k ∣ X = x ) f(x)=argmin_{y \in\gamma}\sum\limits_{k=1}^KL(c_k,y)*P(c_k|X=x)=argmin_{y \in\gamma}\sum\limits_{k=1}^KP(y \ne c_k|X=x) f(x)=argminy∈γk=1∑KL(ck,y)∗P(ck∣X=x)=argminy∈γk=1∑KP(y=ck∣X=x)
= a r g m i n y ∈ γ ( 1 − P ( y = c k ∣ X = x ) ) = a r g m a x y ∈ γ P ( y = c k ∣ X = x ) =argmin_{y \in\gamma}(1-P(y = c_k|X=x))=argmax_{y \in\gamma}P(y = c_k|X=x) =argminy∈γ(1−P(y=ck∣X=x))=argmaxy∈γP(y=ck∣X=x)
所以得到, f ( x ) = a r g m a x y ∈ γ P ( y = c k ∣ X = x ) f(x)=argmax_{y \in\gamma}P(y = c_k|X=x) f(x)=argmaxy∈γP(y=ck∣X=x)
朴素贝叶斯的参数估计
- 极大似然估计
P ( Y = c k ) = ∑ i = 1 N I ( y i = c k ) N , k = 1 , 2 , . . . , K P(Y=c_k)=\frac{\sum\limits_{i=1}^NI(y_i=c_k)}{N},k=1,2,...,K P(Y=ck)=Ni=1∑NI(yi=ck),k=1,2,...,K
P ( X ( j ) = a j l ∣ Y = c k ) = ∑ i = 1 N I ( x i ( j ) = a j l , y i = c k ) ∑ i = 1 N I ( y i = c k ) , j = 1 , 2 , . . . , n ; l = 1 , 2 , . . , S j ; k = 1 , 2 , . . . , K P(X^{(j)}=a_{jl}|Y=c_k)=\frac{\sum\limits_{i=1}^NI(x_i^{(j)}=a_{jl},y_i=c_k)}{\sum\limits_{i=1}^NI(y_i=c_k)},j=1,2,...,n;l=1,2,..,S_j;k=1,2,...,K P(X(j)=ajl∣Y=ck)=i=1∑NI(yi=ck)i=1∑NI(xi(j)=ajl,yi=ck),j=1,2,...,n;l=1,2,..,Sj;k=1,2,...,K, a j l a_{jl} ajl为第 j j j个特征可能取值 - 学习与分类算法
按照极大似然估计统计出来后计算 P ( Y = c k ) ∗ ∏ i = 1 N P ( X ( i ) = x ( i ) ∣ Y = c k ) , k = 1 , 2 , . . . , K P(Y=c_k)*\prod_{i=1}^N P(X^{(i)}=x^{(i)}|Y=c_k),k=1,2,...,K P(Y=ck)∗∏i=1NP(X(i)=x(i)∣Y=ck),k=1,2,...,K
y = f ( x ) = a r g m a x y ∈ γ P ( y = c k ∣ X = x ) y=f(x)=argmax_{y \in\gamma}P(y = c_k|X=x) y=f(x)=argmaxy∈γP(y=ck∣X=x) - 贝叶斯估计
因为直接统计会出现概率为0的现象,这很不合理,出现偏差。这时我们进行平滑处理,
P ( Y = c k ) = ∑ i = 1 N I ( y i = c k ) + λ N + K ∗ λ , k = 1 , 2 , . . . , K P(Y=c_k)=\frac{\sum\limits_{i=1}^NI(y_i=c_k)+\lambda}{N+K*\lambda},k=1,2,...,K P(Y=ck)=N+K∗λi=1∑NI(yi=ck)+λ,k=1,2,...,K
P ( X ( j ) = a j l ∣ Y = c k ) = ∑ i = 1 N I ( x i ( j ) = a j l , y i = c k ) + λ ∑ i = 1 N I ( y i = c k ) + S j ∗ λ , j = 1 , 2 , . . . , n ; l = 1 , 2 , . . , S j ; k = 1 , 2 , . . . , K P(X^{(j)}=a_{jl}|Y=c_k)=\frac{\sum\limits_{i=1}^NI(x_i^{(j)}=a_{jl},y_i=c_k)+\lambda}{\sum\limits_{i=1}^NI(y_i=c_k)+S_j*\lambda},j=1,2,...,n;l=1,2,..,S_j;k=1,2,...,K P(X(j)=ajl∣Y=ck)=i=1∑NI(yi=ck)+Sj∗λi=1∑NI(xi(j)=ajl,yi=ck)+λ,j=1,2,...,n;l=1,2,..,Sj;k=1,2,...,K, a j l a_{jl} ajl
K K K为种类数, S j S_j Sj为属性 j j j的个数
课后实现