python如何读取百万级的csv文件-Python-读取巨大的.csv文件

小编典典

你正在将所有行读入列表，然后处理该列表。不要那样做。

在生成行时对其进行处理。如果需要首先过滤数据，请使用生成器函数：

import csv

def getstuff(filename, criterion):

with open(filename, "rb") as csvfile:

datareader = csv.reader(csvfile)

yield next(datareader) # yield the header row

count = 0

for row in datareader:

if row[3] == criterion:

yield row

count += 1

elif count:

# done when having read a consecutive series of rows

return

我还简化了你的过滤器测试；逻辑相同，但更为简洁。

因为只匹配与条件匹配的单个行序列，所以还可以使用：

import csv

from itertools import dropwhile, takewhile

def getstuff(filename, criterion):

with open(filename, "rb") as csvfile:

datareader = csv.reader(csvfile)

yield next(datareader) # yield the header row

# first row, plus any subsequent rows that match, then stop

# reading altogether

# Python 2: use `for row in takewhile(...): yield row` instead

# instead of `yield from takewhile(...)`.

yield from takewhile(

lambda r: r[3] == criterion,

dropwhile(lambda r: r[3] != criterion, datareader))

return

你现在可以getstuff()直接循环。在getdata()：

def getdata(filename, criteria):

for criterion in criteria:

for row in getstuff(filename, criterion):

yield row

现在直接getdata()在你的代码中循环：

for row in getdata(somefilename, sequence_of_criteria):

# process row

现在，你仅在内存中保留一行，而不是每个条件存储数千行。

2020-02-21