当从不同大小的分布中随机抽样时,我惊讶地发现执行时间似乎主要是根据从中采样的数据集的大小而不是被采样的值的数量来缩放.例:
import pandas as pd
import numpy as np
import time as tm
#generate a small and a large dataset
testSeriesSmall = pd.Series(np.random.randn(10000))
testSeriesLarge = pd.Series(np.random.randn(10000000))
sampleSize = 10
tStart = tm.time()
currSample = testSeriesLarge.sample(n=sampleSize).values
print('sample %d from %d values: %.5f s' % (sampleSize, len(testSeriesLarge), (tm.time() - tStart)))
tStart = tm.time()
currSample = testSeriesSmall.sample(n=sampleSize).values
print('sample %d from %d values: %.5f s' % (sampleSize, len(testSeriesSmall), (tm.time() - tStart)))
sampleSize = 1000
tStart = tm.time()
currSample = testSeriesLarge.sample(n=sampleSize).values
print('sample %d from %d values: %.5f s' % (sampleSize, len(testSeriesLarge), (tm.time() - tStart)))
tStart = tm.time()
currSample = testSeriesSmall.sample(n=sampleSize).values
print('sample %d from %d values: %.5f s' % (sampleSize, len(testSeriesSmall), (tm.time() - tStart)))
输出是:
sample 10 from 10000 values: 0.00126 s
sample 10 from 10000000 values: 1.10504 s
sample 1000 from 10000 values: 0.00122 s
sample 1000 from 10000000 values: 1.15000 s
这似乎违反直觉.也许我很密集,但问题似乎与生成随机索引列表类似,我本来期望采样的数量与数据集的大小和数据集的大小无关.我已经尝试了另外一个或两个具有类似结果的实现,但它开始觉得我只是错过了一个基本问题.
我的问题有两个:(1)这是大pandas的根本问题还是实施的怪癖? (2)以这种方式从大型数据集中随机抽样是否有明显更快的方法?