python3 二进制文件比较快_以定义的格式读取二进制文件的最快方式？

主要更新：修改为使用正确的代码来读取预处理的数组文件(下面的函数using_preprocessed_file())，这大大改变了结果。在

为了确定Python中哪种方法更快(只使用内置函数和标准库)，我创建了一个脚本(通过timeit)来测试可以用于执行此操作的不同技术。它有点长，所以为了避免分心，我只发布代码测试和相关结果。(如果对方法有足够的兴趣，我将发布整个脚本。)

下面是比较的代码片段：@TESTCASE('Read and constuct piecemeal with struct')

def read_file_piecemeal():

structures = []

with open(test_filenames[0], 'rb') as inp:

size = fmt1.size

while True:

buffer = inp.read(size)

if len(buffer) != size: # EOF?

break

structures.append(fmt1.unpack(buffer))

return structures

@TESTCASE('Read all-at-once, then slice and struct')

def read_entire_file():

offset, unpack, size = 0, fmt1.unpack, fmt1.size

structures = []

with open(test_filenames[0], 'rb') as inp:

buffer = inp.read() # read entire file

while True:

chunk = buffer[offset: offset+size]

if len(chunk) != size: # EOF?

break

structures.append(unpack(chunk))

offset += size

return structures

@TESTCASE('Convert to array (@randomir part 1)')

def convert_to_array():

data = array.array('d')

record_size_in_bytes = 9*4 + 16*8 # 9 ints + 16 doubles (standard sizes)

with open(test_filenames[0], 'rb') as fin:

for record in iter(partial(fin.read, record_size_in_bytes), b''):

values = struct.unpack("<2i5d2idi3d2i3didi3d", record)

data.extend(values)

return data

@TESTCASE('Read array file (@randomir part 2)', setup='create_preprocessed_file')

def using_preprocessed_file():

data = array.array('d')

with open(test_filenames[1], 'rb') as fin:

n = os.fstat(fin.fileno()).st_size // 8

data.fromfile(fin, n)

return data

def create_preprocessed_file():

""" Save array created by convert_to_array() into a separate test file. """

test_filename = test_filenames[1]

if not os.path.isfile(test_filename): # doesn't already exist?

data = convert_to_array()

with open(test_filename, 'wb') as file:

data.tofile(file)

在我的系统上运行的结果如下：

^{pr2}$

有趣的是，Python2中的大多数代码片段实际上更快。。。在Fastest to slowest execution speeds using Python 2.7.13

(10 executions, best of 3 repetitions)

Size of structure: 164

Number of structures in test file: 40,000

file size: 6,560,000 bytes

Read array file (@randomir part 2): 0.03586 secs, relative 1.00x ( 0.00% slower)

Read all-at-once, then slice and struct: 0.27871 secs, relative 7.77x ( 677.17% slower)

Read and constuct piecemeal with struct: 0.40804 secs, relative 11.38x (1037.81% slower)

Convert to array (@randomir part 1): 1.45830 secs, relative 40.66x (3966.41% slower)