4.李航机器学习-朴素贝叶斯及Laplace平滑公式笔记与python实现

朴素贝叶斯

项目链接：

公式推导

贝叶斯算法的原理，在李航机器学习的书中已有详细证明，一下只对关键问题进行证明

1 为什么贝叶斯中后验概率最大化等价于经验风险最小化

• 令$L(y,f(x))$为损失函数，通过积分可以得到经验损失
  $$\begin{array}{l}{R}_{\mathrm{e}\mathrm{p}\mathrm{p}}(f)={\int }_x{\int }_{y}L(y,f(x))\times P(x,y)dxdy\\ ={\int }_x{\int }_{y}L(y,f(x))\times P(y|x)P(x)dxdy\\ ={\int }_{x}P(x)dx{\int }_{y}L(y,f(x))\times P(y|x)dy\\ \min \left({\int }_{y}L(y,f(x))\times P(y|x)dy\right)\to \min \left({\sum }_{i=1}^{n}L\left(y_i,f\left(x_i\right)\right)\times P\left(Y=y_i|X=x_i\right)\right)\\ \min \left({\sum }_{k=1}^{K}L\left(c_k,y\right)\times P\left(c_k|X=x\right)\right)\end{array}$$
• 令损失函数为指示函数，则可得等价形式
  $$\begin{array}{l}\min \left({\sum }_{k=1}^{K}L\left(c_k,y\right)\times P\left(c_k|X=x\right)\right)\\ =\min \left({\sum }_{k=1}^{K}P\left(c_k̸=y|X=x\right)\right)\\ =\min \left({\sum }_{k=1}^{K}1-P\left(c_k=y|X=x\right)\right)\\ =\max \left(P\left(c_k=y|X=x\right)\right)\end{array}$$
  2 贝叶斯原理公式

$$P\left(Y=c_k|X=x\right)=\frac{P\left(X=x|Y=c_k\right)P\left(Y=c_k\right)}{{\sum }_{k}P\left(X=x|Y=c_k\right)P\left(Y=c_k\right)}$$

$$P\left(Y=c_k|X=x\right)=\frac{P\left(Y=c_k\right){\prod }_{j}P\left(X^{(j)}={\dot{x}}^{(j)}|Y=c_k\right)}{{\sum }_{k}P\left(Y=c_k\right){\prod }_{j}P\left(X^{(j)}=x^{(j)}|Y=c_k\right)}$$

import numpy as np

class NavieBayesClassifier:
    def __init__(self, lamb = 0):
        self.prior_prob_y = {}
        self.prior_prob_x = {}
        self.x_dim = 0
        #拉普拉斯平滑系数
        self.lamb = lamb
    def fit(self, x, y):
        '''
        x是二维ndarray数组
        y是一维ndarray数组
        x，y长度相同
        '''
        self.x_dim = len(x[0])
        y_list = y.tolist()
        y_unique = np.unique(y)
        for val in y_unique:
            self.prior_prob_y[val] = y_list.count(val)/len(y_list)
        y = np.array([y_list])
        xy = np.hstack((x, y.T))
        for d in range(self.x_dim):
            #处理x不同维度
            x_and_y = xy[:, (d,-1)]
            x_unique = np.unique(xy[:, d])
            laplace = len(x_unique)
            self.prior_prob_x[d] = {}
            for yy in y_unique:
                #处理不同的y值
                x_when_yy = x_and_y[x_and_y[:, -1] == yy]
                x_list = x_when_yy[:, 0].tolist()
                self.prior_prob_x[d][yy] = {}
                for xx in x_unique:
                    #获取固定的y下，不同的x的概率
                    self.prior_prob_x[d][yy][xx] = (x_list.count(xx) + self.lamb) / (len(x_list) + laplace * self.lamb)
    def predict(self, x):
        '''
        x是一维数组
        '''
        res = {}
        all_pro = 0
        for y_val in self.prior_prob_y:
            res[y_val] = self.prior_prob_y[y_val]
            px_y = 1
            for d in range(self.x_dim):
                print(d, y_val, x[d], self.prior_prob_x[d][y_val][x[d]])
                px_y *= self.prior_prob_x[d][y_val][x[d]]
            res[y_val] *= px_y
            all_pro += res[y_val]
        for y_val in res:
            res[y_val] /= all_pro
#利用书中的实例测试
# if __name__ == '__main__':
xy = [[1,4,-1],
    [1,5,-1],
    [1,5,1],
    [1,4,1],
    [1,4,-1],
    [2,4,-1],
    [2,5,-1],
    [2,5,1],
    [2,6,1],
    [2,6,1],
    [3,6,1],
    [3,5,1],
    [3,5,1],
    [3,6,1],
    [3,6,-1]]
xy = np.array(xy)


sb_clf = NavieBayesClassifier(1)
sb_clf.fit(xy[:, (0,1)], xy[:, -1])

print('x prob', sb_clf.prior_prob_x)
print('y prob', sb_clf.prior_prob_y)