python将dataframe导出为csv_python – 将大型DataFrame输出到CSV文件中的最快方法是什么？...

对于

python / pandas我发现df.to_csv(fname)的工作速度为每分钟〜1毫升.有时候我可以提高性能7,像这样：

def df2csv(df,fname,myformats=[],sep=','):

"""

# function is faster than to_csv

# 7 times faster for numbers if formats are specified,

# 2 times faster for strings.

# Note - be careful. It doesn't add quotes and doesn't check

# for quotes or separators inside elements

# We've seen output time going down from 45 min to 6 min

# on a simple numeric 4-col dataframe with 45 million rows.

"""

if len(df.columns) <= 0:

return

Nd = len(df.columns)

Nd_1 = Nd - 1

formats = myformats[:] # take a copy to modify it

Nf = len(formats)

# make sure we have formats for all columns

if Nf < Nd:

for ii in range(Nf,Nd):

coltype = df[df.columns[ii]].dtype

ff = '%s'

if coltype == np.int64:

ff = '%d'

elif coltype == np.float64:

ff = '%f'

formats.append(ff)

fh=open(fname,'w')

fh.write(','.join(df.columns) + '\n')

for row in df.itertuples(index=False):

ss = ''

for ii in xrange(Nd):

ss += formats[ii] % row[ii]

if ii < Nd_1:

ss += sep

fh.write(ss+'\n')

fh.close()

aa=DataFrame({'A':range(1000000)})

aa['B'] = aa.A + 1.0

aa['C'] = aa.A + 2.0

aa['D'] = aa.A + 3.0

timeit -r1 -n1 aa.to_csv('junk1') # 52.9 sec

timeit -r1 -n1 df2csv(aa,'junk3',myformats=['%d','%.1f','%.1f','%.1f']) # 7.5 sec

注意：性能的提高取决于dtypes.

但总是如此(至少在我的测试中)

该to_csv()的执行速度比非优化的python慢​​得多.

如果我有一个4500万行的csv文件,那么：

aa = read_csv(infile) # 1.5 min

aa.to_csv(outfile) # 45 min

df2csv(aa,...) # ~6 min

问题：

What are the ways to make the output even faster?

What's wrong with to_csv() ? Why is it soooo slow ?

注意：我的测试是使用熊猫0.9.1在Linux服务器上的本地驱动器上完成的.