python pandas读取csv指定行_python – pandas中csv的条件行读取

你可以用块读取csv.由于pd.read_csv将在指定chunksize参数时返回迭代器,因此您可以使用itertools.takewhile只读取所需数量的块,而无需读取整个文件.

import itertools as IT

import pandas as pd

chunksize = 10 ** 5

chunks = pd.read_csv(filename, chunksize=chunksize, header=None)

chunks = IT.takewhile(lambda chunk: chunk['B'].iloc[-1] < 10, chunks)

df = pd.concat(chunks)

mask = df['B'] < 10

df = df.loc[mask]

或者,为了避免必须使用df.loc [mask]从最后一个块中删除不需要的行,可能更清晰的解决方案是定义自定义生成器：

import itertools as IT

import pandas as pd

def valid(chunks):

for chunk in chunks:

mask = chunk['B'] < 10

if mask.all():

yield chunk

else:

yield chunk.loc[mask]

break

chunksize = 10 ** 5

chunks = pd.read_csv(filename, chunksize=chunksize, header=None)

df = pd.concat(valid(chunks))