NaiveBayes-参数求解

最新推荐文章于 2022-04-06 16:20:11 发布

维格堂406小队

最新推荐文章于 2022-04-06 16:20:11 发布

阅读量843

点赞数

分类专栏： ★★★机器学习 # ★★分类&回归

本文链接：https://blog.csdn.net/wendaomudong_l2d4/article/details/79570873

版权

★★★机器学习同时被 2 个专栏收录

62 篇文章 3 订阅

订阅专栏

★★分类&回归

23 篇文章 0 订阅

订阅专栏

knitr::opts_chunk$set(echo = TRUE)

朴素贝叶斯直观上倒是很容易理解，无非就是求后验概率最大化，但是损失函数、参数求解都是一知半解。本文以离散型朴素贝叶斯为例，做一些简单的探讨。

前置知识

主要是概率论的一些知识：

条件概率

P (A | B) = P ( A B ) P ( B )

$P\left( {A|B} \right) = \frac{{P\left( {AB} \right)}}{{P\left( B \right)}}$

全概率公式

P (B) = \sum i = 1 n P (A i B) = \sum i = 1 n P (A i) P (B | A i)

$P\left( B \right) = \sum\limits_{i = 1}^n {P\left( {{A_i}B} \right)} = \sum\limits_{i = 1}^n {P\left( {{A_i}} \right)} P\left( {B|{A_i}} \right)$

贝叶斯公式

P (A i | B) = P ( A i B ) P ( B ) = P ( A i ) P ( B | A i ) \sum j = 1 n P ( A j ) P ( B | A i )

$P\left( {{A_i}|B} \right) = \frac{{P\left( {{A_i}B} \right)}}{{P\left( B \right)}} = \frac{{P\left( {{A_i}} \right)P\left( {B|{A_i}} \right)}}{{\sum\limits_{j = 1}^n {P\left( {{A_j}} \right)} P\left( {B|{A_i}} \right)}}$

基本方法

符号标记

设输出空间 $X$ 是 $n$ 维向量的集合，输出空间为类标记集合 $Y=\{c_1,c_2,...,c_k\}$ 。
输入特征： $x \in X$
输出： $y \in Y$ , $y$ 取值为 $c_1,c_2,...,c_K$
联合分布： $P(X,Y)$
训练集： $T=\{(x_1,y_1),...,(x_N,y_N)\}$ ，样本独立同分布
先验分布： $P(Y=c_k)=\theta_k$

条件独立性假设

Naive Bayes中有一个很强且不太符合直觉的假设，类内独立性或者说条件独立性。具体如下：

P (X = x | Y = c k) = P (X (1) = x (1), X (2) = x (2), . . ., X (n) = x (n) | Y = c k) = P (X (1) = x (1) | Y = c k) * P (X (2) = x (2) | Y = c k) * . . . * P (X (n) = x (n) | Y = c k) = \prod j = 1 n P (X (j) = x (j) | Y = c k)

$\begin{array}{l} P\left( {X = x\left| {Y = {c_k}} \right.} \right) = P\left( {{X^{\left( 1 \right)}} = {x^{\left( 1 \right)}},{X^{\left( 2 \right)}} = {x^{\left( 2 \right)}},...,{X^{\left( n \right)}} = {x^{\left( n \right)}}\left| {Y = {c_k}} \right.} \right)\\ = P\left( {{X^{\left( 1 \right)}} = {x^{\left( 1 \right)}}\left| {Y = {c_k}} \right.} \right) * P\left( {{X^{\left( 2 \right)}} = {x^{\left( 2 \right)}}\left| {Y = {c_k}} \right.} \right)*... * P\left( {{X^{\left( n \right)}} = {x^{\left( n \right)}}\left| {Y = {c_k}} \right.} \right)\\ = \prod\limits_{j = 1}^n {P\left( {{X^{\left( j \right)}} = {x^{\left( j \right)}}\left| {Y = {c_k}} \right.} \right)} \end{array}$

目标函数

预测新样本，我们希望求出 $P\left( {Y = {c_k}\left| {X = x} \right.} \right)$ 最大的类 $c_k$ ，作为 $X$ 的类标记，则：

\begin{array}{l} P (Y = c_{k} | X = x) = \frac{P (Y = c_{k}, X = x)}{P (X = x)} \\ = \frac{P (Y = c_{k}, X = x)}{\sum_{k = 1}^{K} P (Y = c_{k}, X = x)} \\ = \frac{P (X = x | Y = c_{k}) P (Y = c_{k})}{\sum_{k = 1}^{K} P (X = x | Y = c_{k}) P (Y = c_{k})} \end{array}

$\begin{array}{l} P\left( {Y = {c_k}\left| {X = x} \right.} \right) = \frac{{P\left( {Y = {c_k},X = x} \right)}}{{P\left( {X = x} \right)}}\\ = \frac{{P\left( {Y = {c_k},X = x} \right)}}{{\sum\limits_{k = 1}^K {P\left( {Y = {c_k},X = x} \right)} }}\\ = \frac{{P\left( {X = x\left| {Y = {c_k}} \right.} \right)P\left( {Y = {c_k}} \right)}}{{\sum\limits_{k = 1}^K {P\left( {X = x\left| {Y = {c_k}} \right.} \right)P\left( {Y = {c_k}} \right)} }} \end{array}$
根据类内独立性：

P (Y = c k | X = x) = P ( Y = c k ) \prod j = 1 n P ( X ( j ) = x ( j ) | Y = c k ) \sum k = 1 K P ( Y = c k ) \prod j = 1 n P ( X ( j ) = x ( j ) | Y = c k )

$P\left( {Y = {c_k}\left| {X = x} \right.} \right) = \frac{{P\left( {Y = {c_k}} \right)\prod\limits_{j = 1}^n {P\left( {{X^{\left( j \right)}} = {x^{\left( j \right)}}\left| {Y = {c_k}} \right.} \right)} }}{{\sum\limits_{k = 1}^K {P\left( {Y = {c_k}} \right)\prod\limits_{j = 1}^n {P\left( {{X^{\left( j \right)}} = {x^{\left( j \right)}}\left| {Y = {c_k}} \right.} \right)} } }}$

因为分母对于所有的 $c_k$ 都是相同的，所以有目标函数：

arg max c k P (Y = c k) \prod j = 1 n P (X (j) = x (j) | Y = c k)

$\arg \;\mathop {\max }\limits_{{c_k}} \;P\left( {Y = {c_k}} \right)\prod\limits_{j = 1}^n {P\left( {{X^{\left( j \right)}} = {x^{\left( j \right)}}\left| {Y = {c_k}} \right.} \right)}$

损失函数

此处大谬，周末过来改一下~2018-04-12
上节的想法是后验概率最大化，那么从损失函数怎么理解呢？
假设朴素贝叶斯的损失函数是 $0-1$ 损失函数：

L (Y, f (X)) = {1, Y \neq f (x) 0, Y = f (x)

$L\left( {Y,f\left( X \right)} \right) = \left\{ {\begin{array}{*{20}{c}} {1,\;\;Y \ne f\left( x \right)}\\ {0,\;\;Y = f\left( x \right)} \end{array}} \right.$
其中

f(X) f ( X ) $f\left( X \right)$ 是分类决策函数，此时期望风险函数为：

R exp (f) = E [L (Y, f (X))] = \sum k = 1 K L (Y, f (X)) P (Y = c k, X = x) = \sum k = 1 K L (Y, f (X)) P (X = x | Y = c k) P (Y = c k)

$\begin{array}{l} {{\mathop{\rm R}\nolimits} _{\exp }}\left( f \right) = E\left[ {L\left( {Y,f\left( X \right)} \right)} \right]\\ = \sum\limits_{k = 1}^K {L\left( {Y,f\left( X \right)} \right)} P\left( {Y = {c_k},X = x} \right)\\ = \sum\limits_{k = 1}^K {L\left( {Y,f\left( X \right)} \right)} P\left( {X = x\left| {Y = {c_k}} \right.} \right)P\left( {Y = {c_k}} \right) \end{array}$
那么损失函数最小化即：

arg m i n c k \sum k = 1 K L (Y, f (X)) P (Y = c k | X = x) P (X = x) = arg m i n c k \sum k = 1 K L (Y, f (X)) P (Y = c k | X = x) = arg m i n c k \sum k = 1 K P (Y \neq c k | X = x) = arg m i n c k (1 - P (Y = c k | X = x)) = arg m a x c k P (Y = c k | X = x)

$\begin{array}{l} \arg \;\mathop {min}\limits_{{c_k}} \sum\limits_{k = 1}^K {L\left( {Y,f\left( X \right)} \right)} P\left( {Y = {c_k}\left| {X = x} \right.} \right)P\left( {X = x} \right)\\ = \arg \;\mathop {min}\limits_{{c_k}} \sum\limits_{k = 1}^K {L\left( {Y,f\left( X \right)} \right)} P\left( {Y = {c_k}\left| {X = x} \right.} \right)\\ = \arg \;\mathop {min}\limits_{{c_k}} \sum\limits_{k = 1}^K {P\left( {Y \ne {c_k}\left| {X = x} \right.} \right)} \\ = \arg \;\mathop {min}\limits_{{c_k}} \left( {1 - P\left( {Y = {c_k}\left| {X = x} \right.} \right)} \right)\\ = \arg \;\mathop {max}\limits_{{c_k}} P\left( {Y = {c_k}\left| {X = x} \right.} \right) \end{array}$

参数求解

回到后验概率最大，我们需要求解 $P(X=x|Y=c_k)$ 和 $P(Y= c_k)$ ，参数求解利用极大似然估计，新增一下若干符号标记：
$n_k$ 是类 $c_k$ 出现的次数
$P(Y=c_k)=\theta_k$ 为先验分布
$x^j$ 为第 $j$ 的特征，其取值为 $a_{jl}$ ，其中 $j=1,2,...J$ ，即特征共 $m$ 维， $l=1,2,...,L$ 即第 $j$ 维的特征共 $L$ 个取值
$P(x^j=a_{jl}|y=c_k)$ 用 $\theta_{c_k}^{a_{jl}}$ 表示
$n_{j,l,k}$ 指第 $k$ 类中第 $j$ 个特征取值为 $a_{jl}$ 的频数
1、似然函数

L = \prod i = 1 n P (x i, y i) = \prod i = 1 n P (y i) P (x i | y i) = \prod k = 1 K (θ k) n k \prod i = 1 n \prod j = 1 J \prod l = 1 L P (x j i = a j l | y i) n j l k = \prod k = 1 K (θ k) n k \prod i = 1 n \prod j = 1 J \prod l = 1 L (θ a j l c k) n j l k

$\begin{array}{l} L = \prod\limits_{i = 1}^n {P\left( {{x_i},{y_i}} \right)} \\ = \prod\limits_{i = 1}^n {P\left( {{y_i}} \right)} P\left( {{x_i}\left| {{y_i}} \right.} \right)\\ = \prod\limits_{k = 1}^K {{{\left( {{\theta _k}} \right)}^{{n_k}}}} {\prod\limits_{i = 1}^n {\prod\limits_{j = 1}^J {\prod\limits_{l = 1}^L {P\left( {x_i^j = {a_{jl}}\left| {{y_i}} \right.} \right)} } } ^{{n_{jlk}}}}\\ = \prod\limits_{k = 1}^K {{{\left( {{\theta _k}} \right)}^{{n_k}}}} {\prod\limits_{i = 1}^n {\prod\limits_{j = 1}^J {\prod\limits_{l = 1}^L {\left( {\theta _{{c_k}}^{{a_{jl}}}} \right)} } } ^{{n_{jlk}}}} \end{array}$
取对数：

log (L) = \sum k = 1 K n k log θ k + \sum i = 1 n \sum j = 1 J \sum l = 1 L n j l k log θ a j l c k

$\log \left( L \right) = \sum\limits_{k = 1}^K {{n_k}\log } {\theta _k} + \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^J {\sum\limits_{l = 1}^L {{n_{jlk}}\log } } } \theta _{{c_k}}^{{a_{jl}}}$
分别记：

f 1 = \sum k = 1 K n k log θ k f 2 = \sum i = 1 n \sum j = 1 J \sum l = 1 L n j l k log θ a j l c k

$\begin{array}{l} {f_1} = \sum\limits_{k = 1}^K {{n_k}\log } {\theta _k}\\ {f_2} = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^J {\sum\limits_{l = 1}^L {{n_{jlk}}\log } } } \theta _{{c_k}}^{{a_{jl}}} \end{array}$
分别求

f1,f2 f 1 , f 2 $f_1,f_2$ 的极值即可，对于

f1 f 1 $f_1$ ,使用拉格朗日乘子法求解：

L 1 = \sum k = 1 K n k log θ k + λ (\sum k = 1 K θ k - 1) ⎧ ⎩ ⎨ ⎪ ⎪ ⎪ ⎪ \partial L 1 \partial θ k = n k θ k - λ = 0 \partial L 2 \partial λ = \sum k = 1 K θ k - 1 = 0 s o : n 1 θ 1 = n 2 θ 2 = \dots = n K θ K = λ a n d : \sum k = 1 K n k = N; \sum k = 1 K θ k = 1 ∴ θ k = n k N

$\begin{array}{l} {L_1} = \sum\limits_{k = 1}^K {{n_k}\log } {\theta _k} + \lambda \left( {\sum\limits_{k = 1}^K {{\theta _k} - 1} } \right)\\ \left\{ {\begin{array}{*{20}{c}} {\frac{{\partial {L_1}}}{{\partial {\theta _k}}} = \frac{{{n_k}}}{{{\theta _k}}} - \lambda = 0}\\ {\frac{{\partial {L_2}}}{{\partial \lambda }} = \sum\limits_{k = 1}^K {{\theta _k} - 1} = 0} \end{array}} \right.\\ so:\\ \frac{{{n_1}}}{{{\theta _1}}} = \frac{{{n_2}}}{{{\theta _2}}} = \cdots = \frac{{{n_K}}}{{{\theta _K}}} = \lambda \\ and:\\ \sum\limits_{k = 1}^K {{n_k}} = N;\sum\limits_{k = 1}^K {{\theta _k}} = 1\\ \therefore {\theta _k} = \frac{{{n_k}}}{N} \end{array}$
对于

f2 f 2 $f_2$ 同样使用拉格朗日乘子法：

L 2 = \sum i = 1 n \sum j = 1 J \sum l = 1 L n j l k log θ a j l c k + λ (\sum k = 1 K θ k - 1) ⎧ ⎩ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ \partial L 1 \partial θ a j l c k = n j l k θ a j l c k - λ = 0 \partial L 2 \partial λ = \sum k = 1 K θ k - 1 = 0 s o : P (x j = a j l | y = c k) = θ a j l c k = n j l k n k = \sum n i = 1 I ( x j i = a j l , y i = c k ) \sum n i = 1 I ( y i = c k )

$\begin{array}{l} {L_2} = \sum\limits_{i = 1}^n {\sum\limits_{j = 1}^J {\sum\limits_{l = 1}^L {{n_{jlk}}\log } } } \theta _{{c_k}}^{{a_{jl}}} + \lambda \left( {\sum\limits_{k = 1}^K {{\theta _k} - 1} } \right)\\ \left\{ {\begin{array}{*{20}{c}} {\frac{{\partial {L_1}}}{{\partial \theta _{{c_k}}^{{a_{jl}}}}} = \frac{{{n_{jlk}}}}{{\theta _{{c_k}}^{{a_{jl}}}}} - \lambda = 0}\\ {\frac{{\partial {L_2}}}{{\partial \lambda }} = \sum\limits_{k = 1}^K {{\theta _k} - 1} = 0} \end{array}} \right.\\ so:\\ P\left( {{x^j} = {a_{jl}}\left| {y = {c_k}} \right.} \right) = \theta _{{c_k}}^{{a_{jl}}} = \frac{{{n_{jlk}}}}{{{n_k}}} = \frac{{\sum\nolimits_{i = 1}^n {I\left( {x_i^j = {a_{jl}},{y_i} = {c_k}} \right)} }}{{\sum\nolimits_{i = 1}^n {I\left( {{y_i} = {c_k}} \right)} }} \end{array}$