有一些问题。我正在对一个160万长度的数据集进行twitter情感分析。由于我的电脑不能做这项工作(由于计算太多),教授让我用大学服务器。在
我刚刚意识到在服务器上,python版本是2.7,它不允许我使用csv reader中的参数encoding来读取文件。在
{否则我不能随时从cd1>中删除其余的数据集。我试着把网站上的所有问题都解决了,但什么也没解决。在
我只想跳过引起错误的那一行,因为集合足够大,可以让我进行很好的分析。在class UTF8Recoder:
def __init__(self, f, encoding):
self.reader = codecs.getreader(encoding)(f)
def __iter__(self):
return self
def next(self):
return self.reader.next().encode("utf-8", errors='ignore')
class UnicodeReader:
def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
f = UTF8Recoder(f, encoding)
self.reader = csv.reader(f, dialect=dialect, **kwds)
def next(self):
'''next() -> unicode
This function reads and returns the next line as a Unicode string.
'''
row = self.reader.next()
return [unicode(s, "utf-8", errors='replace') for s in row]
def __iter__(self):
return self
def extraction(file, textCol, sentimentCol):
"The function reads the tweets"
#fp = open(file, "r",encoding="utf8")
fp = open(file, "r")
tweetreader = UnicodeReader(fp)
#tweetreader = csv.reader( fp, delimiter=',', quotechar='"', escapechar='\\' )
tweets = []
for row in tweetreader:
# It takes the column in which the tweets and the sentiment are
if row[sentimentCol]=='positive' or row[sentimentCol]=='4':
tweets.append([remove_stopwords(row[textCol]), 'positive']);
else:
if row[sentimentCol]=='negative' or row[sentimentCol]=='0':
tweets.append([remove_stopwords(row[textCol]), 'negative']);
else:
if row[sentimentCol]=='irrilevant' or row[sentimentCol]=='2' or row[sentimentCol]=='neutral':
tweets.append([remove_stopwords(row[textCol]), 'neutral']);
tweets = filterWords(tweets)
fp.close()
return tweets;
错误:
^{pr2}$