我试图将数据写入h5py数据集,但使用高内存12核GCE实例写入SSD磁盘,但它运行了13个小时,看不到尽头。
我在GCE实例上运行Jupyter Notebook,在将大量小文件(存储在第二个非ssd磁盘上)添加到存储在ssd磁盘上的文件中的h5py数据集之前,将它们解压最大形状=(29914, 251328)
块=(59, 982)
压缩=gzip
数据类型=float64
我的代码列在下面#Get a sample
minsample = 13300
sampleWithOutReplacement = random.sample(ListOfPickles,minsample)
print(h5pyfile)
with h5py.File(h5pyfile, 'r+') as hf:
GroupToStore = hf.get('group')
datasetToStore = GroupToStore.get('ds1')
#Unpickle the contents and add in th h5py file
for idx,files in enumerate(sampleWithOutReplacement):
#Sample the minimum number of examples
time FilePath = os.path.join(SourceOfPickles,files)
#Use this method to auto close the file
with open(FilePath,"rb") as f:
%time DatasetToStore[idx:] = pickle.load(f)
#print("Processing file ",idx)
print("File Closed")
磁盘上的h5py文件似乎增加了1.4GB,我使用上面和下面的代码填充的每个数据集是我在h5py文件中创建数据集的代码
^{pr2}$
为了减少填充h5py文件所需的时间,我可以对我的配置或代码或两者进行哪些改进?在
更新1
我给jupyter笔记本添加了一些魔法来计时这个过程,我欢迎任何关于加快加载到数据存储的建议,据报道这需要8小时CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 14.1 µs
CPU times: user 8h 4min 11s, sys: 1min 18s, total: 8h 5min 30s
Wall time: 8h 5min 29s