python阅读文件_在Python中阅读巨大的文件

我有一个384MB的文本文件,5000万行.每行包含2个空格分隔的整数：一个键和一个值.该文件按键排序.我需要一种有效的方法来查找

Python中约200个键列表的值.

我目前的做法包括在下面.需要30秒必须有更高效的Python foo才能使这种效率达到几秒钟的合理效率.

# list contains a sorted list of the keys we need to lookup

# there is a sentinel at the end of list to simplify the code

# we use pointer to iterate through the list of keys

for line in fin:

line = map(int, line.split())

while line[0] == list[pointer].key:

list[pointer].value = line[1]

pointer += 1

while line[0] > list[pointer].key:

pointer += 1

if pointer >= len(list) - 1:

break # end of list; -1 is due to sentinel

编码二进制搜索求解(谢谢kigurai！)：

entries = 24935502 # number of entries

width = 18 # fixed width of an entry in the file padded with spaces

# at the end of each line

for i, search in enumerate(list): # list contains the list of search keys

left, right = 0, entries-1

key = None

while key != search and left <= right:

mid = (left + right) / 2

fin.seek(mid * width)

key, value = map(int, fin.readline().split())

if search > key:

left = mid + 1

else:

right = mid - 1

if key != search:

value = None # for when search key is not found

search.result = value # store the result of the search