tensorflow读取npz数据以及shuffle Filling up shuffle buffer降低性能

https://stackoverflow.com/questions/48889482/feeding-npy-numpy-files-into-tensorflow-data-pipeline
https://www.tensorflow.org/guide/data#applying_arbitrary_python_logic_with_tfpy_func

Feeding .npy (numpy files) into tensorflow data pipeline

You can do it with tf.py_func, see the example here. The parse function would simply decode the filename from bytes to string and call np.load.

Update: something like this:

def read_npy_file(item):
    data = np.load(item.decode())
    return data.astype(np.float32)

file_list = ['/foo/bar.npy', '/foo/baz.npy']

dataset = tf.data.Dataset.from_tensor_slices(file_list)

dataset = dataset.map(
        lambda item: tuple(tf.py_func(read_npy_file, [item], [tf.float32,])))

https://www.coder.work/article/4991774

python - Tensorflow Dataset API改组将性能降低9倍

降低shuffle 时的buffersize， 但目前好像batchsize越大也会导致shuffle变慢