效率差异: 集合直接使用 add方法 与 判断后再用集合的 add方法

背景:

• 一个有2百多万行的测试文件，按行统计得出不重复行内容及各自的数量. 集合 lineset = set()
• 再读取每一行后 linetmp = f.readline()，使用4个方法处理。见下面的代码
• win10, anaconda 4.8.3, python 3.8.3

结果

• test_add_if() 用的时间最短。
• 也就是, 不了解效率的情况下，使用一般的判断流程图，即可得到最佳效率的小程序。(对很多数据来说。数据少了，也没人在乎效率了)

#
#  comparing time difference between set().add(newitem) and
#  if newitem not in list, then add to set().add(newitem)
#  conclusion: test_add_if() is the best way for 2M+ lines check.
#

test_set = set()
test_file = "test_set_if.dxf"


def set_add_directly():
    global test_set
    with open(test_file, 'r') as f:
        line_tmp = f.readline()
        while line_tmp:
            test_set.add(line_tmp)
            line_tmp = f.readline()


def set_add_if():
    global test_set
    with open(test_file,'r') as f:
        line_tmp = f.readline()
        while line_tmp:
            if line_tmp not in test_set:
                test_set.add(line_tmp)
            line_tmp = f.readline()


def set_line_split():
    global test_set
    with open(test_file, 'r') as f:
        lines = f.readlines()
        linelist = [line.split for line in lines]
        test_set = set(linelist)
    #  print('set function:', test_set, flush=True)


def set_f(): # there is \n for each element. more time is needed.
    global test_set
    with open(test_file,'r') as f:
        test_set = set (f.readlines())


# main
if __name__ == '__main__':
    from timeit import Timer
    timer1 = Timer('set_add_directly()', 'from __main__ import set_add_directly')
    t1 = timer1.timeit(1) # one round is more than 1s. so it is timeit(1) not 10000.
    timer2 = Timer('set_add_if()', 'from __main__ import set_add_if')
    t2 = timer2.timeit(1)
    timer3 = Timer('set_line_split()', 'from __main__ import set_line_split')
    t3 = timer3.timeit(1)
    timer4 = Timer('set_f()', 'from __main__ import set_f')
    t4 = timer4.timeit(1)
    print('set_add_directly - set_add_if:', t1-t2, flush=True)
    print('set_add_directly - set_line_split:', t1-t3, flush=True)
    print('set_line_split - set_add_if:', t3-t2, flush=True)
    print('set_f - set_add_if:', t4-t2, flush=True)

• 其中一次的结果:
  ---------- Python ----------
  set_add_directly - set_add_if: 0.06032050000000011
  set_add_directly - set_line_split: -0.24342030000000003
  set_line_split - set_add_if: 0.30374080000000014
  set_f - set_add_if: 0.10101979999999977

Output completed (3 sec consumed) - Normal Termination