python对csv去除重复行_使用python脚本从csv文件中删除重复的行

更新：2016

如果您乐意使用有用的more_itertools外部库：

from more_itertools import unique_everseen

with open('1.csv','r') as f, open('2.csv','w') as out_file:

out_file.writelines(unique_everseen(f))

@ IcyFlame解决方案的更有效的版本

with open('1.csv','r') as in_file, open('2.csv','w') as out_file:

seen = set() # set for fast O(1) amortized lookup

for line in in_file:

if line in seen: continue # skip duplicate

seen.add(line)

out_file.write(line)

要在同一个文件中编辑,您可以使用它

import fileinput

seen = set() # set for fast O(1) amortized lookup

for line in fileinput.FileInput('1.csv', inplace=1):

if line in seen: continue # skip duplicate

seen.add(line)

print line, # standard output is now redirected to the file