python读取大文件csv_在python中读取大csv文件行

^{}不会将整个文件读入内存。当您迭代reader对象时，它会懒洋洋地逐行迭代该文件。所以你可以像平常一样使用reader，但是在你读了很多你想读的行之后，从你的迭代中得到{Initializer for the reader objecT:

static PyObject *

csv_reader(PyObject *module, PyObject *args, PyObject *keyword_args)

{

PyObject * iterator, * dialect = NULL;

ReaderObj * self = PyObject_GC_New(ReaderObj, &Reader_Type);

if (!self)

return NULL;

self->dialect = NULL;

self->fields = NULL;

self->input_iter = NULL;

self->field = NULL;

// stuff we dont care about here

// ...

self->input_iter = PyObject_GetIter(iterator); // here we save the iterator (file object) we passed in

if (self->input_iter == NULL) {

PyErr_SetString(PyExc_TypeError,

"argument 1 must be an iterator");

Py_DECREF(self);

return NULL;

}

static PyObject *

Reader_iternext(ReaderObj *self) // This is what gets called when you call `next(reader_obj)` (which is what a for loop does internally)

{

PyObject *fields = NULL;

Py_UCS4 c;

Py_ssize_t pos, linelen;

unsigned int kind;

void *data;

PyObject *lineobj;

if (parse_reset(self) < 0)

return NULL;

do {

lineobj = PyIter_Next(self->input_iter); // Equivalent to calling `next(input_iter)`

if (lineobj == NULL) {

/* End of input OR exception */

if (!PyErr_Occurred() && (self->field_len != 0 ||

self->state == IN_QUOTED_FIELD)) {

if (self->dialect->strict)

PyErr_SetString(_csvstate_global->error_obj,

"unexpected end of data");

else if (parse_save_field(self) >= 0)

break;

}

return NULL;

}

如您所见，next(reader_object)在内部调用next(file_object)。所以你要逐行遍历这两个，而不是把整个东西读入内存。在