python vectorize_bigrams python的CountVectorize词汇规范

你的问题是因为你的字典2是基于元组的。这是一个极简主义的例子，它表明当bigram是字符串时，这是有效的。如果要分别处理每个文件，可以将其传递给矢量器.transform()作为列表。在from sklearn.feature_extraction.text import CountVectorizer

Doc1 = 'Wimbledon is one of the four Grand Slam tennis tournaments, the others being the Australian Open, the French Open and the US Open.'

Doc2 = 'Since the Australian Open shifted to hardcourt in 1988, Wimbledon is the only major still played on grass'

doc_set = [Doc1, Doc2]

my_vocabulary= ['Grand Slam', 'Australian Open', 'French Open', 'US Open']

vectorizer = CountVectorizer(ngram_range=(2, 2))

vectorizer.fit_transform(my_vocabulary)

term_count = vectorizer.transform(doc_set)

# Show the index key for each bigram

vectorizer.vocabulary_

Out[11]: {'grand slam': 2, 'australian open': 0, 'french open': 1, 'us open': 3}

# Sparse matrix of bigram counts - each row corresponds to a document

term_count.toarray()

Out[12]:

array([[1, 1, 1, 1],

[1, 0, 0, 0]], dtype=int64)

你可以使用列表理解来修改你的词典2。在

^{pr2}$

编辑：基于以上，我认为你可以使用以下代码：from sklearn.feature_extraction.text import CountVectorizer

# Modify dictionary2 to be compatible with CountVectorizer

dictionary2_cv = [' '.join(tup) for tup in dictionary2]

# Initialize and train CountVectorizer

cv2 = CountVectorizer(ngram_range=(2, 2))

cv2.fit_transform(dictionary2_cv)

for row in range(start,end+1):

report_name = fund_reports_table.loc[row, "report_names"]

raw_report = open("F:/EDGAR_ShareholderReports/" + report_name, 'r', encoding="utf8").read()

## word for word

temp = cv1.fit_transform([raw_report]).toarray()

res1 = np.concatenate((res1,temp),axis=0)

## big grams

bigram=set()

sentences = raw_report.split(".")

for line in sentences:

token = nltk.word_tokenize(line)

bigram = bigram.union(set(list(ngrams(token, 2))) )

# Modify bigram to be compatible with CountVectorizer

bigram = [' '.join(tup) for tup in bigram]

# Note you must not fit_transform here - only transform using the trained cv2

temp = cv2.transform(list(bigram)).toarray()

res2=np.concatenate((res2,temp),axis=0)