NLP(II):使用NLTK进行数据预处理
在上一节中我们使用原始的正则表达式对语料进行处理,显然,这是费力不讨好的工作。本着“不重复造轮子”的原则,这节开始我们用现成的轮子,调包侠出击。
原始数据准备
使用tweepy获得有关football和fishing的推文各一万条,具体讲解见上一节。代码如下:
注意:bearer_token用你自己的,别用我的。
import tweepy
client = tweepy.Client(bearer_token='AAAAAAAAAAAAAAAAAAAAACZyhAEAAAAAwq%2ByW0tqZ4TbIbtKlDx3w9dpghc%3DG7qdk7bQvPvl5r9W5fKXhLAx4SFSiOD32I5kIZjVISOF79F2WG')
query = 'fishing lang:en -has:mentions -has:links -is:retweet' # tweets have 'nft', written in English, not retwee, do not have links
tweets = list(tweepy.Paginator(client.search_recent_tweets, query=query, tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(limit=10000))
print("{} tweets are collected.".format(len(tweets)))
import csv
driveFolderDirectory = './'
savedFileName = 'tweetsFishing.csv'
pathToSave = driveFolderDirectory + savedFileName
with open(pathToSave, 'w', newline='', encoding='utf-8') as csvfile:
fieldnames = ['idx','tweetId', 'createdTime', 'tweetText']
writer = csv.DictWriter(csvfile, fi