python输出名字和字数,用Python计算字数

I have a list of strings in python.

list = [ "Sentence1. Sentence2...", "Sentence1. Sentence2...",...]

I want to remove stop words and count occurrence of each word of all different strings combined. Is there a simple way to do it?

I am currently thinking of using CountVectorizer() from scikit and than iterating for each word and combining the results

解决方案

If you don't mind installing a new python library, I suggest you use gensim.

The first tutorial does exactly what you ask:

# remove common words and tokenize

stoplist = set('for a of the and to in'.split())

texts = [[word for word in document.lower().split() if word not in stoplist]

for document in documents]

You will then need to create the dictionary for your corpus of document and create the bag-of-words.

dictionary = corpora.Dictionary(texts)

dictionary.save('/tmp/deerwester.dict') # store the dictionary, for future

print(dictionary)

You can weight the result using tf-idf and stuff and do LDA quite easily after.

Have a look at the tutorial 1 here