Naive Bayes

【参数定义】

先验概率分布，$P\left ( y=1 \right )=\phi_y$

条件概率分布，参数共$2n$个
$\begin{matrix}P\left ( x_1=1|y=1 \right )=\phi_{1|y=1} & P\left ( x_1=1|y=0 \right )=\phi_{1|y=1} \\ P\left ( x_2=1|y=1 \right )=\phi_{2|y=1} & P\left ( x_2=1|y=0 \right )=\phi_{2|y=1} \\ \vdots & \vdots \\ P\left ( x_n=1|y=1 \right )=\phi_{n|y=1} & P\left ( x_n=1|y=0 \right )=\phi_{n|y=1}\end{matrix}$

【条件独立性假设】

$\begin{aligned}P(x|y)&=P(x_1,x_2,\cdots,x_n|y)\\&=P(x_1|y)P(x_2|y,x_1)P(x_3|y,x_1,x_2)\cdots P(x_n|y,x_1,x_2,\cdots,x_n)\\&=\prod_{i=1}^{n}P(x_i|y)\end{aligned}$

【参数估计】

$\begin{aligned}\phi_y=\frac{\sum_\limits{i=1}^{m}\mathbf{1}\left \{ y^{(i)}=1 \right \}}{m}\end{aligned}$

$\begin{aligned}\phi_{j|y=1}=\frac{\sum_\limits{i=1}^{m}\mathbf{1}\left \{ x_j^{(i)}=1\wedge y^{(i)}=1 \right \}}{\sum_\limits{i=1}^{m}\mathbf{1}\left \{ y^{(i)}=1 \right \}}\end{aligned}$

$\begin{aligned}\phi_{j|y=1}=\frac{\sum_\limits{i=1}^{m}\mathbf{1}\left \{ x_j^{(i)}=1\wedge y^{(i)}=0 \right \}}{\sum_\limits{i=1}^{m}\mathbf{1}\left \{ y^{(i)}=0 \right \}}\end{aligned}$

【预测未知样本$\hat{x}$，需要计算后验概率】

$\begin{aligned}P\left ( y=1|\hat{x} \right )&=\frac{P\left ( \hat{x},y=1 \right )}{P\left ( \hat{x} \right )}\\&=\frac{P(y=1)P\left ( \hat{x}|y=1 \right )}{P\left ( \hat{x} \right )}\\&\propto P(y=1)P\left ( \hat{x}|y=1 \right )\\&=P(y=1)\prod_{i=1}^{n}P\left ( \hat{x}_i|y=1 \right )\end{aligned}$

同理
$\begin{aligned}P\left ( y=0|\hat{x} \right )\propto P(y=0)\prod_{i=1}^{n}P\left ( \hat{x}_i|y=0 \right )\end{aligned}$

如果一定要计算分子$P\left ( \hat{x} \right )$

$\begin{aligned}P\left ( \hat{x} \right )&=\sum_{y}P\left ( \hat{x},y \right )\\&=\sum_{y}P(y)P\left ( \hat{x}|y \right )\\&=P(y=1)P\left ( \hat{x}|y=1 \right )+P(y=0)P\left ( \hat{x}|y=0 \right )\\&=P(y=1)\prod_{i=1}^{n}P\left ( \hat{x}_i|y=1 \right )+P(y=0)\prod_{i=1}^{n}P\left ( \hat{x}_i|y=0 \right )\end{aligned}$