python利用eval方法提升dataframe运算性能

最新推荐文章于 2024-07-24 09:30:00 发布

genghaihua

最新推荐文章于 2024-07-24 09:30:00 发布

阅读量2k

点赞数

分类专栏：机器学习 python

本文链接：https://blog.csdn.net/genghaihua/article/details/92836106

版权

机器学习同时被 2 个专栏收录

37 篇文章 8 订阅

订阅专栏

python

20 篇文章 1 订阅

订阅专栏

eval方法可以直接利用c语言的速度，而不用分配中间数组，不需要中间内存的占用.

如果包含多个步骤，每个步骤都要分配一块内存

import numpy as np
import pandas as pd
import timeit


df = pd.DataFrame({'a': np.random.randn(10000000),
                   'b': np.random.randn(10000000),
                   'c': np.random.randn(10000000),
                   'x': 'x'})
# print df
start_time = timeit.default_timer()
df['a']/( df['b']+0.1)-df['c']
end_time = timeit.default_timer()
print (end_time - start_time)
print "___________________"
start_time = timeit.default_timer()
pd.eval("df['a']/( df['b']+0.1)-df['c']")
end_time = timeit.default_timer()
print (end_time - start_time)

运行时间对比

0.136633455546
___________________
0.087637596342

As of version 0.13 (released January 2014), Pandas includes some experimental tools that allow you to directly access C-speed operations without costly allocation of intermediate arrays.