python 文本去重

最新推荐文章于 2024-06-27 16:41:53 发布

KongJingRou

最新推荐文章于 2024-06-27 16:41:53 发布

阅读量2.5k

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/KongJingRou/article/details/80747487

版权

python 专栏收录该内容

25 篇文章 0 订阅

订阅专栏

# -*- coding: utf8 -*-
#==============================
file_name = 'uk_urls.txt'
#==============================

open('quchong.txt','w').truncate()

with open(file_name, 'r') as f: 
    lines = f.readlines() 
print 'lines = ' + str(len(lines))

x = 0

while x < len(lines):

    file = open('quchong.txt', 'r')
    quchong = file.read()
    file.close()

    if lines[x] not in quchong:
        print '[' + str(x+1) + ']' + 'ok, add'
        file = open('quchong.txt', 'a')
        file.write(lines[x])
        file.close
    else:
        print '[' + str(x+1) + ']' + 'no, del:' + lines[x].replace('\n','')

    x= x + 1