python读取大文件(GB级别)采用的办法很简单:
with open(...) as f:
for line in f:
<do something with line>
例如:
with open(filepath,'r') as infile:
for line in infile:
print line
一切都交给python解释器处理,读取效率很高,且占用资源少。
ref:How to read large file, line by line in python - Stack Overflow