希望分别使用 CountVectorizer 和 TfidfVectorizer 对新闻文本数据进行特征提取，并验证提取是否正确

最新推荐文章于 2023-07-19 16:59:16 发布

immenselee

最新推荐文章于 2023-07-19 16:59:16 发布

阅读量1.6k

点赞数

分类专栏： Python

本文链接：https://blog.csdn.net/immenselee/article/details/104713886

版权

Python 专栏收录该内容

21 篇文章 5 订阅

订阅专栏

编程要求
本关任务，希望分别使用 CountVectorizer 和 TfidfVectorizer 对新闻文本数据进行特征提取，并验证提取是否正确。

数据介绍：

采用fetch_20newsgroups("./step5/",subset='train', categories=categories)函数加载对应目录的新闻数据，subset='train'指定采用的是该数据的训练集，categories指定新闻目录，具体的函数调用可参考官方文档fetch_20newsgroups。X 是一个 string 数组，包含 857 个新闻文档。

本关具体分成为几个子任务：

1.使用 CountVectorizer 方法提取新闻数据的特征向量，返回词汇表大小和前五条特征向量；

要补充的代码块如下：

def transfer2CountVector():
    '''
    使用CountVectorizer方法提取特征向量，返回词汇表大小和前五条特征向量
    返回值:
    vocab_len - 标量，词汇表大小
    tokenizer_list - 数组，对测试字符串test_str进行分词后的结果
    '''
    vocab_len = 0
    test_str = "what's your favorite programming language?"
    tokenizer_list = []
    #   请在此添加实现代码   #
    # ********** Begin *********#
    # ********** End **********#
    return vocab_len,tokenizer_list
2.使用 TfidfVectorizer 方法得到新闻数据的词汇-特征向量映射关系，指定使用内建的英文停止词列表作为停止词，并且词出现的最小频数等于 2。然后，将向量化转换器应用到新的测试数据。

def transfer2TfidfVector():
    '''
        使用TfidfVectorizer方法提取特征向量，并将向量化转换器应用到新的测试数据
        TfidfVectorizer()方法的参数设置：
        min_df = 2,stop_words="english"
        test_data - 需要转换的原数据
        返回值:
        transfer_test_data - 二维数组ndarray
        '''
    test_data = ['Once again, to not believe in God is different than saying....... where is your evidence for that "god is" is meaningful at some level?\n   Benedikt\n']
    transfer_test_data = None
    #   请在此添加实现代码   #
    # ********** Begin *********#
    # ********** End **********#
    return transfer_test_data

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
categories = [
    'alt.atheism',
    'talk.religion.misc',
]
# 加载对应目录的新闻数据，包含857 个文档
data = fetch_20newsgroups("./step5/",subset='train', categories=categories)
X = data.data
def transfer2CountVector():
    '''
    使用CountVectorizer方法提取特征向量，返回词汇表大小和前五条特征向量
    返回值:
    vocab_len - 标量，词汇表大小
    tokenizer_list - 数组，对测试字符串test_str进行分词后的结果
    '''
    vocab_len = 0
    test_str = "what's your favorite programming language?"
    tokenizer_list = []
    #   请在此添加实现代码   #
    # ********** Begin *********#
    vectorizer = CountVectorizer()
    vectorizer.fit(X)
    analyze = vectorizer.build_analyzer()
    tokenizer_list = analyze(test_str)
    vocab_len = len(vectorizer.vocabulary_)
    # ********** End **********#
    return vocab_len,tokenizer_list
def transfer2TfidfVector():
    '''
        使用TfidfVectorizer方法提取特征向量，并将向量化转换器应用到新的测试数据
        TfidfVectorizer()方法的参数设置：
        min_df = 2,stop_words="english"
        返回值:
        transfer_test_data - 矩阵，TFIDF向量化后的数据
        '''
    test_data = ['Once again, to not believe in God is different than saying\n>I BELIEVE that God does not exist. I still maintain the position, even\n>after reading the FAQs, that strong atheism requires faith.\n>\n \nNo it in the way it is usually used. In my view, you are saying here that\ndriving a car requires faith that the car drives.\n \nFor me it is a conclusion, and I have no more faith in it than I have in the\npremises and the argument used.\n \n \n>But first let me say the following.\n>We might have a language problem here - in regards to "faith" and\n>"existence". I, as a Christian, maintain that God does not exist.\n>To exist means to have being in space and time. God does not HAVE\n>being - God IS Being. Kierkegaard once said that God does not\n>exist, He is eternal. With this said, I feel it\'s rather pointless\n>to debate the so called "existence" of God - and that is not what\n>I\'m doing here. I believe that God is the source and ground of\n>being. When you say that "god does not exist", I also accept this\n>statement - but we obviously mean two different things by it. However,\n>in what follows I will use the phrase "the existence of God" in it\'s\n>\'usual sense\' - and this is the sense that I think you are using it.\n>I would like a clarification upon what you mean by "the existence of\n>God".\n>\n \nNo, that\'s a word game. The term god is used in a different way usually.\nWhen you use a different definition it is your thing, but until it is\ncommonly accepted you would have to say the way I define god is ... and\nthat does not exist, it is existence itself, so I say it does not exist.\n \nInterestingly, there are those who say that "existence exists" is one of\nthe indubitable statements possible.\n \nFurther, saying god is existence is either a waste of time, existence is\nalready used and there is no need to replace it by god, or you are implying\nmore with it, in which case your definition and your argument so far\nare incomplete, making it a fallacy.\n \n \n(Deletion)\n>One can never prove that God does or does not exist. When you say\n>that you believe God does not exist, and that this is an opinion\n>"based upon observation", I will have to ask "what observtions are\n>you refering to?" There are NO observations - pro or con - that\n>are valid here in establishing a POSITIVE belief.\n(Deletion)\n \nWhere does that follow? Aren\'t observations based on the assumption\nthat something exists?\n \nAnd wouldn\'t you say there is a level of definition that the assumption\n"god is" is meaningful. If not, I would reject that concept anyway.\n \nSo, where is your evidence for that "god is" is meaningful at some level?\n   Benedikt\n']
    transfer_test_data = None
    #   请在此添加实现代码   #
    # ********** Begin *********#
    tfidf_vertor = TfidfVectorizer(min_df=2, stop_words="english")
    tfidf_vertor.fit(X)
    transfer_test_data = tfidf_vertor.transform(test_data).toarray()
    # ********** End **********#
    return transfer_test_data