python排除词汇库_Python TfidfVectorizer throw:空词汇;也许文件只包含停用词“

我正在尝试使用

Python的Tfidf来转换文本语料库.

但是,当我尝试fit_transform它时,我得到一个值错误ValueError:空词汇;也许这些文件只包含停用词.

In [69]: TfidfVectorizer().fit_transform(smallcorp)

---------------------------------------------------------------------------

ValueError Traceback (most recent call last)

in ()

----> 1 TfidfVectorizer().fit_transform(smallcorp)

/Users/maxsong/anaconda/lib/python2.7/site-packages/sklearn/feature_extraction/text.pyc in fit_transform(self, raw_documents, y)

1217 vectors : array, [n_samples, n_features]

1218 """

-> 1219 X = super(TfidfVectorizer, self).fit_transform(raw_documents)

1220 self._tfidf.fit(X)

1221 # X is already a transformed view of raw_documents so

/Users/maxsong/anaconda/lib/python2.7/site-packages/sklearn/feature_extraction/text.pyc in fit_transform(self, raw_documents, y)

778 max_features = self.max_features

779

--> 780 vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)

781 X = X.tocsc()

782

/Users/maxsong/anaconda/lib/python2.7/site-packages/sklearn/feature_extraction/text.pyc in _count_vocab(self, raw_documents, fixed_vocab)

725 vocabulary = dict(vocabulary)

726 if not vocabulary:

--> 727 raise ValueError("empty vocabulary; perhaps the documents only"

728 " contain stop words")

729

ValueError: empty vocabulary; perhaps the documents only contain stop words

我在这里阅读了SO问题:Problems using a custom vocabulary for TfidfVectorizer scikit-learn并尝试了ogrisel建议使用TfidfVectorizer(** params).build_analyzer()(dataset2)来检查文本分析步骤的结果,这似乎按预期工作:下面的代码段:

In [68]: TfidfVectorizer().build_analyzer()(smallcorp)

Out[68]:

[u'due',

u'to',

u'lack',

u'of',

u'personal',

u'biggest',

u'education',

u'and',

u'husband',

u'to',

还有别的我做错了吗?我正在喂它的语料库只是一条由换行符打断的巨大长串.

谢谢!

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值