程序跑的很慢,使用bottleneck一分析发现问题出在deepcopy上。
因为程序是需要对列表进行操作,而我的程序逻辑是要求必须对列表进行深拷贝,于是没有办法绕过。
只能想办法加速这个深层拷贝。
我想到四种方法
第一种:deepcopy,这没得啥说的。是baseline.
##直接deepcopy
def method1(origin_list, step):
for each in range(step):
l = copy.deepcopy(origin_list)
return l
第二种:使用numpy,先转为numpy对象,然后再tolist
##转换为numpy, 然后再tolist()
def method2(origin_list, step):
for each in range(step):
l = np.array(origin_list).tolist()
assert type(l) == type(origin_list)
return l
第三种:使用pickle,先pickle.dump,再pickle.load
##使用pickle
def method3(origin_list, step):
for each in range(step):
l = pickle.loads(pickle.dumps(origin_list))
assert type(l)== type(origin_list)
return l
第四种,使用ujson,也是先usjon.dump,在ujson.load。
##使用ujson
def method4(origin_list, step):
for each in range(step):
l = ujson.loads(ujson.dumps(origin_list))
assert type(l)== type(origin_list)
return l
分别拷贝10000次,测试如下:
__author__ = 'dk'
import numpy as np
import copy
import time
import pickle
import ujson
##直接deepcopy
def method1(origin_list, step):
for each in range(step):
l = copy.deepcopy(origin_list)
return l
##转换为numpy, 然后再tolist()
def method2(origin_list, step):
for each in range(step):
l = np.array(origin_list).tolist()
assert type(l) == type(origin_list)
return l
##使用pickle
def method3(origin_list, step):
for each in range(step):
l = pickle.loads(pickle.dumps(origin_list))
assert type(l)== type(origin_list)
return l
##使用ujson
def method4(origin_list, step):
for each in range(step):
l = ujson.loads(ujson.dumps(origin_list))
assert type(l)== type(origin_list)
return l
if __name__ == '__main__':
origin_list = [[i for i in range(30)] for i in range(30)]
step = 10000
t = time.time()
method1(origin_list,step)
e = time.time()
print(e-t)
t = time.time()
method2(origin_list,step)
e = time.time()
print(e-t)
t = time.time()
method3(origin_list,step)
e = time.time()
print(e-t)
t = time.time()
method4(origin_list,step)
e = time.time()
print(e-t)
输出结果:
方法 | 耗时 |
---|---|
deepcopy | 6.8964385986328125 |
numpy | 1.1347527503967285 |
pickle | 0.5621602535247803 |
ujson | 0.7545688152313232 |
所以,如果list深层次拷贝绕不过去,就用pickle吧!
哎~·