python对csv去除重复行_使用python脚本从csv文件中删除重复的行

5 个答案:

答案 0 :(得分：48)

更新：2016

如果您愿意使用有用的more_itertools外部库：

from more_itertools import unique_everseen

with open('1.csv','r') as f, open('2.csv','w') as out_file:

out_file.writelines(unique_everseen(f))

@ IcyFlame解决方案的更有效版本

with open('1.csv','r') as in_file, open('2.csv','w') as out_file:

seen = set() # set for fast O(1) amortized lookup

for line in in_file:

if line in seen: continue # skip duplicate

seen.add(line)

out_file.write(line)

要就地编辑同一个文件，您可以使用此

import fileinput

seen = set() # set for fast O(1) amortized lookup

for line in fileinput.FileInput('1.csv', inplace=1):

if line in seen: continue # skip duplicate

seen.add(line)

print line, # standard output is no