http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer
class sklearn.feature_extraction.text. CountVectorizer ( input=u'content', encoding=u'utf-8', decode_error=u'strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, token_pattern=u'(?u)\b\w\w+\b', ngram_range=(1, 1), analyzer=u'word', max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<type 'numpy.int64'> ) [source]
作用:Convert a collection of text documents to a matrix of token counts(计算词汇的数量,即tf);结果由 scipy.sparse.coo_matrix进行稀疏表示。
看下参数就知道CountVectorizer在提取tf时都做了什么:
strip_accents : {‘ascii’, ‘unicode’, None}:是否除去“音调”,不知道什么是“音调”?看:http://textmechanic.com/?reqp=1&reqr=nzcdYz9hqaSbYaOvrt==
lowercase : boolean, True by default:计算tf前,先将所有字符转化为小写。这个参数一般为True。
preprocessor : callable or None (default):复写the preprocessing (string transformation) stage,但保留tokenizing and n-grams generation steps.这个参数可以自己写。