python中单词汇总_从python中的单词列表中单词单词化？

my program has list of words and amongst that i need few specific words to be tokenized as one word.

my program would split a string into words eg

str="hello my name is vishal, can you please help me with the red blood cells and platelet count. The white blood cell is a single word."

output will be

list=['hello','my','name','is','vishal','can','you','please','help','me','with','the','red','blood','cells','and','platelet','count','the','white','blood','cell','is','a','single','word']..

now I want is to tokenize words such as 'red blood cell' as a single word. There are many such words in my list that has three or more words to be considered as one such as 'platelet count','white blood cell',etc. any suggestion for doing that.

解决方案

from nltk.tokenize import MWETokenizer

str="hello my name is vishal, can you please help me with the red blood cells and platelet count. The white blood cell is a single word."

tokenizer = MWETokenizer()

tokenizer.add_mwe(('red', 'blood', 'cells'))

tokenizer.add_mwe(('white','blood','cell'))

tokenizer.add_mwe(('platelet','count.'))

print(tokenizer.tokenize(str.split()))

Output

`['hello', 'my', 'name', 'is', 'vishal,', 'can', 'you', 'please', 'help', 'me', 'with', 'the', 'red_blood_cells', 'and', 'platelet_count.', 'The', 'white_blood_cell', 'is', 'a', 'single', 'word.']`

You can use the Multi-Word Expression Tokenizer MWETokenizer