python3 文本文件内容去除重复行

最新推荐文章于 2020-12-18 00:25:06 发布

whatday

最新推荐文章于 2020-12-18 00:25:06 发布

阅读量2.3k

点赞数

原文链接：https://blog.csdn.net/whatday/article/details/104874321

版权

环境：python3.8

import shutil

# 文件去除重复行
def remove_duplicates(path):
    lines_seen = set()
    outfile = open(f"{path}.out", 'a+', encoding='utf-8')
    f = open(path, 'r', encoding='utf-8')
    for line in f:
        if line not in lines_seen:
            outfile.write(line)
            lines_seen.add(line)
    outfile.close()
    f.close()

    # 移动文件
    shutil.move(f"{path}.out", path)