中文垃圾邮件分类。2种特征提取，词袋特征，IFIDF分布特征，贝叶斯NB,LR,SVM各自表现的实战示例代码

最新推荐文章于 2024-03-18 18:30:47 发布

光英的记忆

最新推荐文章于 2024-03-18 18:30:47 发布

阅读量2.4k

点赞数 5

分类专栏： tensorflow NLP 文章标签：中文垃圾邮件分类 NB LR SVM

本文链接：https://blog.csdn.net/qq_29678299/article/details/93606192

版权

该博客详细介绍了如何进行中文垃圾邮件分类，通过词袋模型和TF-IDF分布特征进行特征提取，使用朴素贝叶斯(NB)、逻辑回归(LR)和支持向量机(SVM)三种模型进行训练，并进行了预测与效果评估。数据集包括ham_data.txt和spam_data.txt，同时排除了stop_words.utf8中的停用词。

摘要由CSDN通过智能技术生成

结论：

数据：

ham_data.txt

spam_data.txt

stop_words.utf8

数据处理：

"""

@author: liushuchun
"""
import re
import string
import jieba

# 加载停用词
with open("dict/stop_words.utf8", encoding="utf8") as f:
    stopword_list = f.readlines()


def tokenize_text(text):
    tokens = jieba.cut(text)
    tokens = [token.strip() for token in tokens]
    return tokens


def remove_special_characters(text):
    tokens = tokenize_text(text)
    pattern = re.compile('[{}]'.format(re.escape(string.punctuation)))
    filtered_tokens = filter(None, [pattern.sub('', token) for token in tokens])
    filtered_text = ' '.join(filtered_tokens)
    return filtered_text


def remove_stopwords(text):
    tokens = tokenize_text(text)
    filtered_tokens = [token for token in tokens if token not in stopword_list]
    filtered_text = ''.join(filtered_tokens)
    return filtered_text


def normalize_corpus(corpus, tokenize=False):
    normalized_corpus = []
    for text in corpus:

        text = remove_special_characters(text)
        text = remove_stopwords(text)
        normalized_corpus.append(text)
        if tokenize:
            text = tokenize_text(text)
            normalized_corpus.append(text)

    return normalized_corpus

特征提取

"""

@author: liushuchun
"""

from sklearn.feature_extraction.text import CountVectorizer


def bow_extractor(corpus, ngram_range=(1, 1)):
    vectorizer = CountVectorizer(min_df=1, ngram_range=ngram_range)
    features = vectorizer.fit_transform(corpus)
    return vectorizer, features


from sklearn.feature_extraction.text import TfidfTransformer


def tfidf_transformer(bow_matrix):
    transformer = TfidfTransformer(norm='l2',
                                   smooth_idf=True,
                                   use_idf=True)
    tfidf_matrix = transformer.fit_transform(bow_matrix)
    return transformer, tfidf_matrix


from sklearn