python pickle文件大小_关于性能：如何减少在python中加载pickle文件所花费的时间...

我用python创建了一个字典，并将其放入pickle中。它的大小达到了300MB。现在，我想装同样的泡菜。

output = open('myfile.pkl', 'rb')

mydict = pickle.load(output)

装这个泡菜大约需要15秒钟。我该如何减少这一时间？

硬件规格：Ubuntu 14.04，4GB RAM

下面的代码显示了使用json、pickle和cpickle转储或加载文件所需的时间。

转储后，文件大小将在300MB左右。

import json, pickle, cPickle

import os, timeit

import json

mydict= {all values to be added}

def dump_json():

output = open('myfile1.json', 'wb')

json.dump(mydict, output)

output.close()

def dump_pickle():

output = open('myfile2.pkl', 'wb')

pickle.dump(mydict, output,protocol=cPickle.HIGHEST_PROTOCOL)

output.close()

def dump_cpickle():

output = open('myfile3.pkl', 'wb')

cPickle.dump(mydict, output,protocol=cPickle.HIGHEST_PROTOCOL)

output.close()

def load_json():

output = open('myfile1.json', 'rb')

mydict = json.load(output)

output.close()

def load_pickle():

output = open('myfile2.pkl', 'rb')

mydict = pickle.load(output)

output.close()

def load_cpickle():

output = open('myfile3.pkl', 'rb')

mydict = pickle.load(output)

output.close()

if __name__ == '__main__':

print"Json dump:"

t = timeit.Timer(stmt="pickle_wr.dump_json()", setup="import pickle_wr")

print t.timeit(1),'

'

print"Pickle dump:"

t = timeit.Timer(stmt="pickle_wr.dump_pickle()", setup="import pickle_wr")

print t.timeit(1),'