python csv 大文件_Python 从大型csv文件中提取感兴趣的行

帮妹子处理一个2.xG 大小的 csv文件，文件太大，不宜一次性读入内存，可以使用open迭代器。

with open(filename,'r') as file

# 按行读取

for line in file:

process

或者简单点

for line in open('myfile.txt','r'):

pass

需求是，提取时间在指定时间段的数据，另存一个文件。

全部代码如下

def is_between_time(str, start, end):

"""

:param str: a line in data file : 8684496663,粤BC5948,2016-01-01 22:01:56,114.083448,22.531582,225,0,0,0,114075022530,114070022530,114.078316,22.534267,1463910,2016-01-01 22:25:59.772000

:param start: start point for example: 21:57:00

:param end: end point for example: 22:03:00

:return:

"""

fields = str.split(',')

datetime = fields[2]

time = datetime.split(' ')[1]

if time > start and time < end:

return True

else:

return False

file_to_read_path = "E:/P_CZCGPS_20160101.csv"

file_to_write = open("E:/result.csv", 'w')

# read file and process

with open(file_to_read_path,'r') as file:

for line in file:

if is_between_time(line, "21:57:00", "22:03:00"):

print(line)

file_to_write.write(line)

file_to_write.close()

1024节日快乐！