python如何读取百万级的csv文件-在python中读取大型csv文件的行

我有一个非常大的csv文件,我无法在内存中完整加载.所以我想逐个阅读,将其转换为numpy数组,然后再进行一些处理.

但问题是它是一个普通的读者,我无法在csvReader中找到任何指定大小的选项.

此外,因为我想将行转换为numpy数组,我不想读取任何一半的行,所以不是指定大小,我想要一些我可以在读者中指定"no of rows”的东西.

有没有内置功能或简单的方法来做到这一点.

最佳答案

csv.reader不会将整个文件读入内存.当您遍历读者对象时,它会逐行地遍历文件.所以你可以像往常一样使用阅读器,但是在你阅读之后要从你的迭代中断,无论你想读多少行.您可以在

C-code used to implement the reader object中看到这一点.

Initializer for the reader objecT:

static PyObject *

csv_reader(PyObject *module, PyObject *args, PyObject *keyword_args)

{

PyObject * iterator, * dialect = NULL;

ReaderObj * self = PyObject_GC_New(ReaderObj, &Reader_Type);

if (!self)

return NULL;

self->dialect = NULL;

self->fields = NULL;

self->input_iter = NULL;

self->field = NULL;

// stuff we dont care about here

// ...

self->input_iter = PyObject_GetIter(iterator); // here we save the iterator (file object) we passed in

if (self->input_iter == NULL) {

PyErr_SetString(PyExc_TypeError,

"argument 1 must be an iterator");

Py_DECREF(self);

return NULL;

}

static PyObject *

Reader_iternext(ReaderObj *self) // This is what gets called when you call `next(reader_obj)` (which is what a for loop does internally)

{

PyObject *fields = NULL;

Py_UCS4 c;

Py_ssize_t pos, linelen;

unsigned int kind;

void *data;

PyObject *lineobj;

if (parse_reset(self) < 0)

return NULL;

do {

lineobj = PyIter_Next(self->input_iter); // Equivalent to calling `next(input_iter)`

if (lineobj == NULL) {

/* End of input OR exception */

if (!PyErr_Occurred() && (self->field_len != 0 ||

self->state == IN_QUOTED_FIELD)) {

if (self->dialect->strict)

PyErr_SetString(_csvstate_global->error_obj,

"unexpected end of data");

else if (parse_save_field(self) >= 0)

break;

}

return NULL;

}

如您所见,next(reader_object)在内部调用next(file_object).所以你要逐行迭代,而不是将整个内容读入内存.