Python文本分析及预处理

文本分析的基本功能学习，包括句子切分、单词切分、大小写转化、删除停用词、题干提取、词性还原。
基本功能学习

#句子切分，单词切分
import nltk
a='i have a pen, i have a apple'
s=nltk.sent_tokenize(a)
print (s)
w=[]
for i in s:
    for j in nltk.word_tokenize(i):
        w.append(j)
        
print (w)

#大小写转换
w='china'
print(w.lower())
print(w.upper())

#删除停用词

#nltk.download('stopwords')

stopwords=nltk.corpus.stopwords.words('english')

#print(stopwords)
a2='May you have enough happiness to make. you love HIM'
w2=[nltk.word_tokenize(i) for i in nltk.sent_tokenize