数据挖掘 实验作业-二元分类器，朴素贝叶斯

Intro

In this tutorial we will implement a Naive Bayes classifier for text classification. More specifically, the aim of the classifier is to predict whether a given product review is positive or negative.

We will represent documents (i.e. product reviews) as vectors of binary features. For each unigram or bigram that appears in at least one document in our training dataset we introduce a binary feature. For a given document a feature is 1 if the documents contains the corresponding unigram/bigram, and it is 0 otherwise.

The datasets are contained in four files (‘test.negative’, ‘test.positive’, ‘train.negative’, ‘train.positive’). The files correspond to product reviews from the test and train datasets labelled positively or negatively. For example, the file ‘test.negative’ contains negative product reviews from the test dataset.

---

The reviews are preprocessed and expressed as a list of unigrams and bigrams without duplications (for a bigram (w1,w2), the corresponding record is w1__w2). Each review is represented as a single line and features (unigrams and bigrams) extracted from that review are listed space delimited. Take a moment to inspect the content of the dataset files.

import numpy as np

1 读取训练集

we read the training data, compute feature statistics, and store it in a dictionary featureStat , where featureStat[feature][classLabel] is equal to the number of documents in the class classLabel that contain feature . We also compute the total number of positive train instances and the total number of negative train instances.

def count(featureStat, fname, classLabel):
    numInstances = 0
    with open(fname) as file:
        for line in file:
            numInstances += 1
            for feature in line.strip().split():
                if feature not in featureStat:
                    featureStat[feature] = {0:0, 1:0}
                featureStat[feature][classLabel] += 1
    return numInstances
featureStat = {}
numPositiveInstances = count(featureStat, "train.positive", 1)
numNegativeInstances = count(featureStat, "train.negative", 0)
print(numPositiveInstances,numNegativeInstances)

2 训练：为每一个出现过的单词，计算正面，负面的可能性

Now, compute the conditional probabilities P(wi=1|C=c) , i.e. probability of specific feature wi to be present in a document from class c . Use Laplace smooting to avoid zero probabilities.

conditional_probs = {}
for wi in featureStat:
    Prob_0= float(featureStat[wi][0]+1)/float(numNegativeInstances+len(featureStat))
    Prob_1= float(featureStat[wi][1]+1)/float(numPositiveInstances+len(featureStat))
    conditional_probs[wi]={0: Prob_0, 1: Prob_1}

print(conditional_probs)

# print(featureStat['atmosphere'])

# print(conditional_probs['atmosphere'][0])

3 创建贝叶斯分类器

Implement a Naive Bayes classfier function that predicts whether a given document belongs to the positive or the negative class. To avoid problems of very small numbers, instead of computing P(C=c)⋅∏iP(wi=ai|C=c) cosider computing the log of that.

import math 

def NaiveBayes(doc):
  log_prob_0 = 0
  log_prob_1 = 0
  length=float(len(featureStat))
  len_log=math.log(length)
  for wi in doc:
    if wi in featureStat:
      Product_0=conditional_probs[wi][0]
      Product_1=conditional_probs[wi][1]
    else:
      Product_0=float(1)/float(numPositiveInstances+len(featureStat))
      Product_1=float(1)/float(numPositiveInstances+len(featureStat))
    log_prob_0 += math.log(Product_0)-len_log
    log_prob_1 += math.log(Product_1)-len_log
  Prob_0 =log_prob_0+math.log(float(numNegativeInstances))-len_log
  Prob_1 =log_prob_1+math.log(float(numPositiveInstances))-len_log
  # print("Probabilistic")
  # print(Prob_0,Prob_1)
  return 0 if Prob_0 > Prob_1 else 1

5 读取测试集数据

Let’s now read the test dataset from the files

def getInstances(fname, classLabel):
    data = []
    with open(fname) as file:
        for line in file:
            data.append((classLabel, line.strip().split()))
    return data

# Read test data

test_data = getInstances("test.positive", 1)
test_data.extend(getInstances("test.negative", 0))

6 使用accuracy 评估该分类器

Evaluate accuracy of the Naive Bayes algorithm on the test data

def NaiveBayes_accuracy(test_data):
  length=len(test_data)
  class_predict = {}
  for (y,doc) in test_data:
    if y not in class_predict:
      class_predict[y] = {0:0, 1:0}
    class_predict[y][NaiveBayes(doc)]+= 1
  # print(class_predict)
  sum_result = class_predict[0][0]+class_predict[1][1]
  # print(sum_result)
  return float(sum_result)/float(length)

print(NaiveBayes_accuracy(test_data))