Dataframe处理速度测试

一、定义一个50000行的Dataframe

a1 = np.random.randint(1,100,[1,50000]).T
a2 = np.random.rand(1,50000).T
c = np.hstack((a1,a2))
df = pd.DataFrame(c,columns=['m1','m2'])

二、定义一个函数

def simple_fun(v):
    return (v**2 - v) // 2 + (v**0.5) // 2

三、测试

1、先取行再取列，平均4.7264s

start = time.time()
m3 = []
for i in range(len(df)):
    m3.append(simple_fun(df.iloc[i]['m1']))
df['m3'] = m3
print(time.time() - start)

2、先取列再取行，0.5956s

start = time.time()
m3 = []
for i in range(len(df)):
    m3.append(simple_fun(df['m1'][i]))
df['m3'] = m3
print(time.time() - start)

3、使用apply，平均0.0409s

start = time.time()
df['m3'] = df['m1'].apply(simple_fun)
print(time.time() - start)

4、使用Pandas series，平均0.0249s

start = time.time()
df['m3'] = simple_fun(df['m1'])
print(time.time() - start)

5、使用NumPy arrays，平均0.0029s

start = time.time()
df['m3'] = simple_fun(df['m1'].values)
print(time.time() - start)