pandas 读取大文件

最新推荐文章于 2024-07-29 02:33:55 发布

Addmana

最新推荐文章于 2024-07-29 02:33:55 发布

阅读量3.3k

点赞数

分类专栏： python 文章标签： pandas

本文链接：https://blog.csdn.net/Addmana/article/details/78493992

版权

python 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

from matplotlib import pyplot  
import scipy as sp  
import numpy as np  
from matplotlib import pylab  
from pandas import Series, DataFrame
import pandas as pd
filename = 'data.txt'
data = pd.read_csv(filename, sep='::', names=list('uirt'), index_col=False, usecols=range(len('uirt')),engine = 'python', iterator=True)
loop = True
chunkSize = 100000
chunks = []
while loop:
    try:
        chunk = data.get_chunk(chunkSize)
        chunks.append(chunk)
    except StopIteration:
        loop = False
        print "Iteration is stopped."
data = pd.concat(chunks, ignore_index=True)